Forward Pass Metrics Trace Reference

Forward Pass Metrics (FPM) tracing persists finalized FPM payloads immediately before the Rust publication path sends them to the event plane. Persistence does not replace or reroute event-plane publication.

Configuration

The configuration variables do not enable tracing by themselves. An invalid value or unwritable output path disables tracing only for that worker and emits a warning. Inference and normal FPM publication continue.

DYN_FORWARDPASS_METRIC_PORT remains a separate legacy backend-generation opt-in and takes precedence when set. It cannot add a missing Dynamo relay to an unsupported topology. DYN_FPM_BENCHMARK_OUTPUT_PATH is benchmark-only and is not used for live tracing.

Supported Topologies

For vLLM, an explicit DYN_FORWARDPASS_METRIC_PORT wins; otherwise trace activation uses port 20380. SGLang uses its existing per-worker IPC endpoint. TensorRT-LLM publishes in-process, so the port value is unused, but setting the variable still opts the worker in.

Capture Modes

In sampled mode, Dynamo retains the newest pending payload for each (namespace, component, worker_id, dp_rank) key. At each monotonic interval it writes keys whose counter_id changed. It does not repeat an unchanged counter, and it flushes pending values during graceful shutdown.

In full mode, Dynamo writes every valid payload, including idle heartbeats. Each writer flushes a non-empty batch every second and can flush sooner when its 1 MiB buffer fills. Producer enqueueing is nonblocking in both modes; a bounded queue drops trace records instead of delaying inference.

File and Retention Semantics

The default files are:

/tmp/dynamo-fpm.<producer-id>.000000.jsonl.gz
/tmp/dynamo-fpm.<producer-id>.000001.jsonl.gz

The producer ID is the sanitized runtime connection ID. Rotation counts uncompressed JSONL bytes. A single oversized row is written intact to an otherwise empty segment. After rolling, Dynamo removes only the oldest files that exactly match that producer’s prefix. On restart, the next index is one greater than the highest matching index.

DYN_FPM_MAX_SEGMENTS applies independently to every producer, including producer IDs left by old worker instances.

Capacity Planning

The roll threshold counts uncompressed JSONL bytes; disk usage is the compressed gzip size. Estimate uncompressed daily volume with:

average JSONL row bytes * records per second * 86400

At the default five-second interval, one continuously changing rank writes about 17,280 periodic rows per day, plus a possible shutdown flush. A 600-byte row is about 10.4 MB per rank per day before compression. In full mode, the same row at 10 forward passes per second is about 518 MB per rank per day.

Record Schema

Each line uses the shared gzip JSONL envelope:

{
  "timestamp": 1250,
  "event": {
    "schema": "dynamo.fpm.trace.v1",
    "source": {
      "namespace": "default",
      "component": "backend",
      "producer_id": "4192"
    },
    "capture_mode": "sampled",
    "observed_at_unix_ms": 1782777601250,
    "fpm": {
      "version": 1,
      "worker_id": "4192",
      "dp_rank": 0,
      "counter_id": 42,
      "wall_time": 0.025,
      "scheduled_requests": {
        "num_prefill_requests": 2,
        "sum_prefill_tokens": 256,
        "sum_prefill_kv_tokens": 64,
        "num_decode_requests": 3,
        "sum_decode_kv_tokens": 1024
      },
      "queued_requests": {
        "num_prefill_requests": 1,
        "sum_prefill_tokens": 128,
        "num_decode_requests": 0,
        "sum_decode_kv_tokens": 0
      }
    }
  }
}

observed_at_unix_ms is the absolute observation time. The outer timestamp is milliseconds since the writer started. The nested fpm object is the canonical payload. A counter_id gap can result from sampling, local queue pressure, an upstream ZMQ drop, a crash, or node loss.

Consuming the Raw FPM Stream

_FpmPublisherThread publishes each ForwardPassMetrics message on a ZMQ PUB socket as a three-frame multipart (self._pub.send_multipart((topic, seq_bytes, payload), flags=zmq.NOBLOCK)) with an empty topic frame (topic = b""), an 8-byte big-endian sequence number (seq_bytes = seq.to_bytes(8, "big")), and the payload last. The payload is MessagePack, not the JSON used by the trace schema above: msgspec.msgpack.Encoder().encode(...) on the same ForwardPassMetrics struct from dynamo.common.forward_pass_metrics. Subscribe with an empty topic filter to receive every message: ZMQ SUB topic matching is a byte-prefix match against the first frame, and an empty filter matches unconditionally.

import zmq
from dynamo.common.forward_pass_metrics import decode
ctx = zmq.Context.instance()
sock = ctx.socket(zmq.SUB)
sock.connect("tcp://127.0.0.1:20380")  # base port + dp_rank
sock.setsockopt(zmq.SUBSCRIBE, b"")
while True:
    topic, seq_bytes, payload = sock.recv_multipart()
    seq = int.from_bytes(seq_bytes, "big")
    metrics = decode(payload)  # ForwardPassMetrics, or None on an unrecognized schema version

Before encoding, the publisher stamps counter_id with its own send-time sequence number (metrics = msgspec.structs.replace(metrics, counter_id=seq)), the same value carried in the sequence frame: it is a per-publisher sequence, not a group step id, stamped from one itertools.count() per _FpmPublisherThread, and it drifts across data-parallel ranks because each rank’s publisher counts from 0 independently. An idle publisher emits a heartbeat once per second, which also advances the sequence. Sends are non-blocking and are dropped rather than queued when the ZMQ send buffer is full, so this raw stream’s counter_id can show the same kind of gap already described for the trace schema above. The self-benchmark (--benchmark-mode) is a separate path: while it runs, this publisher is paused and forward-pass metrics go into the benchmark artifact instead of the socket, with counter_id set to the benchmark point id rather than a publisher sequence; see FPM self-benchmark collection and Engine Provenance in the Self-Benchmark Artifact below.

Each data-parallel rank binds DYN_FORWARDPASS_METRIC_PORT + dp_rank, defaulting to base port 20380. Under attention data parallelism (dp_size > 1), the self-benchmark synchronizer additionally binds DYN_FORWARDPASS_METRIC_PORT + dp_size, but only rank 0 (the data-parallel master) binds it, as a ROUTER socket that every other rank’s DEALER socket connects to as a client. The occupied block (the base port alone at dp_size <= 1, or through base + dp_size once the synchronizer also binds) must sit outside the kernel’s ephemeral port range, which Dynamo reads from /proc/sys/net/ipv4/ip_local_port_range (commonly starting at 32768). A port inside that range can be claimed by an unrelated outbound connection before the publisher binds it, so the bind then fails with Address already in use intermittently and only under load. Dynamo checks this once per engine-core process and logs one warning when the block overlaps the range, staying silent when the proc file cannot be read, such as on a non-Linux host: FPM ports <first>-<last> overlap this host's ephemeral port range <low>-<high>; an outbound connection can take one of them first and the publisher bind then fails intermittently. Set DYN_FORWARDPASS_METRIC_PORT to a base below <low>.

Measurement Protocol in the Self-Benchmark Artifact

Self-benchmark artifacts (--benchmark-mode) record a top-level measurement_protocol block and a benchmark_measurement block inside each retained FPM. The artifact’s schema_version remains 2; both blocks have their own schema_version: 1. These fields describe benchmark input construction, timing reduction, and observed execution evidence. They do not appear in live-serving traces.

Prompt construction combines DYN_BENCH_CONTENT_SEED (default "0") with the point’s coordinates, data-parallel rank, and request slot. Selecting a smaller grid or changing benchmark point order does not change that content identity. A decode point recorded with giant_fake_off_by_batch in its sample_reasons keeps the prompts built for its declared coordinate, whose total_kv_read_tokens exceeds the recorded value by batch_size; its benchmark_measurement.point_key names the recorded coordinate. Different data-parallel ranks intentionally use different content. Compare the same rank’s prompt hashes across runs rather than requiring every rank to have the same hash. Pin the model, tokenizer, dataset, content mode, and pool tag when comparing measurements; see FPM self-benchmark collection.

The shared protocol contains:

Each retained FPM’s benchmark_measurement contains:

The merger requires identical shared protocols in all contributing rank files. It rejects a mixture of legacy artifacts and artifacts with a protocol, and requires point/rank evidence for every member of a synchronized iteration group. Per-rank prompt hashes, raw timings, and execution observations are preserved unchanged; their values need not agree across ranks. Legacy runs without these metadata blocks remain readable, but the merger does not invent evidence for them.

Per-Sample Timing and Graph Dispatch

Each benchmark_measurement.raw_fpms entry identifies one raw scheduler observation. Its benchmark_sample block has this shape:

{
  "sample_index": 0,
  "forward_index": 12,
  "timing": {
    "basis": "schedule_to_output",
    "start_monotonic": 150.0,
    "end_monotonic": 150.062
  },
  "cudagraph": {
    "status": "observed",
    "runtime_mode": "PIECEWISE",
    "num_unpadded_tokens": 80,
    "num_padded_tokens": 80,
    "num_paddings": 0
  }
}

sample_index is the zero-based position in raw_fpms. forward_index counts nonempty forward outputs during this rank’s benchmark, including preparation forwards. Both indices are local to that rank’s process; they do not synchronize ranks.

The timestamps use monotonic-clock seconds in the local process, so timestamps from different ranks cannot be subtracted or ordered against each other. An unavailable start timestamp is null. wall_time retains the native scheduler measurement and existing reduction; these observations do not replace it with GPU execution time. For a reduced estimate, consult estimate.raw_sample_indices and the selected raw observations. The retained estimate has no benchmark_sample, because a median can combine multiple executions.

In benchmark mode, Dynamo enables vLLM’s existing cudagraph_metrics option when the runtime exposes it and it is not already on. Before the worker registers for serving, Dynamo turns an option it enabled back off in the model workers and for vLLM’s stat loggers, and discards the dispatch statistics the loggers collected during the benchmark, so serving neither records nor logs them. An option you enabled yourself stays on. To reach the model workers, Dynamo sets --worker-extension-cls to its own extension class in benchmark mode unless you set one. With your own class, Dynamo skips the two calls that need its extension class, the cudagraph_metrics reset and the worker probe, and each skipped call logs one warning that names your class. The model workers keep recording the statistics, but nothing collects them. The per-sample fields come from ModelRunnerOutput.cudagraph_stats, without adding GPU timing or synchronization. runtime_mode records NONE, PIECEWISE, or FULL; the token counts describe that dispatch’s padding. Runtimes or model runners without complete statistics emit status: "unavailable" and null values. Missing evidence never means NONE.

Observed dispatch is separate from the point’s expected_cudagraph_mode and expected_capture_size, and from the engine’s resolved graph configuration. A mode and padded token count do not identify the complete graph descriptor, a capture-versus-replay event, or the attention kernel that ran. Equal values alone do not establish equivalent execution.

Warmup Completion Evidence

Each rank artifact’s warmup_evidence contains status (recorded or unavailable) and an ordered records list. The list covers global warmup, discarded eager-shape replicas, real-prefix seeding, and real-prefix same-shape preparation. An empty recorded list means none of these attempts was recorded. Decode real-KV warmup, by shared prefill chains or native exact-context stages, is not in this list: each stage is recorded in kvwarm.stages, and kvwarm.initialization_strategy is native_exact_context when the native strategy is selected.

The record count grows with warmup attempts; each attempt retains at most two scheduler shapes. The merged artifact keeps these lists under rank_warmup_evidence, keyed by DP rank. A rank whose artifact was not loaded has explicit unavailable evidence. Compare each raw sample’s forward_index with the same rank’s warmup ranges to establish ordering. A completed request without shape validation remains validation.status: "not_performed"; neither completion nor a shape-validation pass proves matching cache contents, graph preparation, or kernel history.

Engine Provenance in the Self-Benchmark Artifact

A self-benchmark run (--benchmark-mode) writes one JSON artifact per data-parallel rank plus one merged artifact, both separate from the dynamo.fpm.trace.v1 records above. Each carries an optional top-level engine block, captured when the instrumented scheduler starts. Dynamo does not rewrite these artifacts after the benchmark: the post-run worker probe records the resolved worker configuration in a separate sidecar file, and per-forward dispatch belongs to the raw samples described above. The artifact’s schema_version stays 2. Artifacts whose capture never ran, including older artifacts, omit engine.

The following example shows selected fields from a rank artifact. The null and "pending_worker_probe" values are placeholders that stay in the artifact; the worker probe’s results are in the sidecar.

{
  "engine": {
    "attention": {
      "backend_requested": null,
      "backend_per_kind": {},
      "mla_prefill_backend_requested": null,
      "flash_attn_version": 3,
      "use_trtllm_attention": false,
      "indexer_kv_dtype": "fp8",
      "indexer_kv_dtype_configured": "auto",
      "backend_resolved": null,
      "mla_prefill_backend_resolved": null,
      "resolution": "pending_worker_probe",
      "indexer": {
        "index_topk": 2048,
        "index_n_heads": 1,
        "index_head_dim": 128
      }
    },
    "kv_cache": {
      "cache_dtype": "auto",
      "block_size": 64,
      "enable_prefix_caching": true,
      "groups": [
        {
          "type": "FullAttentionSpec",
          "dtype": "bfloat16",
          "head_size": 128,
          "num_kv_heads": 8,
          "block_size": 64,
          "kv_quant_mode": "NONE"
        }
      ],
      "kv_cache_layout": "LBHNC"
    },
    "quantization": {
      "method": "fp8",
      "checkpoint_config": {
        "quant_method": "fp8",
        "activation_scheme": "dynamic"
      },
      "quant_config_class": "Fp8Config"
    },
    "scheduler": {
      "max_num_batched_tokens": 8192,
      "max_num_seqs": 256,
      "async_scheduling": true,
      "enable_chunked_prefill": true,
      "long_prefill_token_threshold": 0
    },
    "model": {
      "dtype": "bfloat16",
      "model_type": "deepseek_v3",
      "architectures": ["DeepseekV3ForCausalLM"],
      "model": "deepseek-ai/DeepSeek-V3"
    },
    "parallel": {
      "data_parallel_size": 4,
      "data_parallel_rank": 0,
      "tensor_parallel_size": 1,
      "pipeline_parallel_size": 1,
      "prefill_context_parallel_size": 1,
      "enable_expert_parallel": true,
      "enable_eplb": false,
      "all2all_backend": "deepep_high_throughput",
      "eplb_config": {
        "window_size": 1000,
        "step_interval": 3000,
        "num_redundant_experts": 0,
        "log_balancedness": false,
        "log_balancedness_interval": 1,
        "use_async": true,
        "policy": "default",
        "communicator": null
      }
    },
    "speculative": null,
    "graph": {
      "cudagraph_mode": "FULL",
      "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 32],
      "resolution": "pre_resolution"
    },
    "versions": {
      "vllm": "0.29.0",
      "vllm_build_commit": "abcdef0",
      "dynamo": "0.9.0",
      "python": "3.12.3"
    },
    "resolved": null,
    "resolution": "pending_worker_probe"
  }
}

• attention records what the engine-core process asked for. vLLM resolves the real attention backend inside the model workers and never writes it back into the engine-core config, so backend_requested and mla_prefill_backend_requested stay null whenever selection was automatic, and backend_per_kind stays {} unless a per-attention-kind override was set. indexer is present only for sparse-indexer models; indexer_kv_dtype is the resolved value and indexer_kv_dtype_configured is the raw configured one ("auto" resolves to the DeepSeek indexer default, "fp8"). backend_resolved, mla_prefill_backend_resolved, and this sub-block’s own resolution are null, null, and "pending_worker_probe" in every artifact; the worker probe described below records their resolved values in the sidecar.
• kv_cache.groups has one entry per KV-cache group the workers reported (type is the spec class name, plus dtype, head_size, num_kv_heads, block_size, and kv_quant_mode when set). num_gpu_blocks is deliberately not here: it is decided per engine-core process after profiling with no cross-rank reduction, so it is rank-varying data rather than engine identity, and it is already recorded per rank at the artifact’s limits.num_gpu_blocks. kv_cache_layout is present starting vLLM 0.29.0 and simply absent on 0.28.0, which keeps it in a worker-process global the engine-core process cannot read. When vLLM reports a group as a UniformTypeKVCacheSpecs wrapper (several same-kind specs that differ only in per-layer attributes), the group’s own dtype, head_size, and num_kv_heads stay null, and the entry additionally carries a specs list with one entry of this same shape per wrapped spec.
• quantization.checkpoint_config is the normalized quantization dict the checkpoint itself declared, not the user-facing online-quantization spec (model_config.quantization_config, deliberately not read here). method is the kernel family vLLM resolved, and quant_config_class is that resolved config object’s class name.
• scheduler and model are read directly from the engine’s own scheduler_config and model_config at startup and need no further resolution. model.max_model_len is deliberately not here: vLLM’s --max-model-len -1 auto-fit can resolve it differently per rank depending on each rank’s free GPU memory, so it is rank-varying data rather than engine identity, and it is already recorded per rank at the artifact’s limits.max_model_len.
• parallel.data_parallel_rank differs per rank and is excluded from the startup identity comparison. The post-run probe results described below are recorded after the merge, in the sidecar, so they are not startup identity evidence.
• speculative is null unless speculative decoding is configured, in which case it is a four-field summary: method, model, num_speculative_tokens, draft_tensor_parallel_size.
• graph freezes what the engine-core process believed about CUDA graphs before the model worker re-resolved them, tagged "resolution": "pre_resolution". Under a multiprocess executor the engine-core process never sees that rewrite, so this sub-block is never updated after capture. Post-resolution values belong to the individual worker-probe replies in the sidecar, not this startup block.
• versions is a best-effort snapshot (vllm, vllm_build_commit, dynamo, python); a single lookup failure leaves just that one key null rather than failing the whole block.
• resolved and resolution are null and "pending_worker_probe" in every artifact whose engine capture completed; the worker probe sidecar holds their post-run values.
• capture_error appears only when the capture function itself raised partway through; it sits alongside whatever sub-blocks were already built, and a capture failure never costs the run its measurements.
• The merged artifact keeps one copy of this block. Every contributing rank’s startup block must be identical except parallel.data_parallel_rank. The comparison excludes post-run fields: resolved, resolved_scope, resolution, worker_probe, and resolved fields under attention. A disagreement in the compared startup fields fails the merge. The merged copy’s parallel.data_parallel_rank is set to null, since the merged document describes every rank at once. A rank whose own capture failed or is missing never blocks the merge; each such rank is instead excluded from the cross-rank identity check and listed, by rank number, in a capture_errors map added to the merged block, valued with that rank’s own capture error or the literal string missing engine block.

Worker Probe Sidecar

After collection, Dynamo calls the fpm_engine_probe method of its worker extension class in every model worker, with one collective_rpc under a 30-second timeout. In benchmark mode Dynamo sets --worker-extension-cls to that class unless you set one; with your own class Dynamo skips the call and the probe results record probe_skipped: .... Dynamo writes the replies, or the reason there are none, to one JSON file next to the merged artifact, named after it with a _worker_probe suffix: benchmark_results_merged.json gets benchmark_results_merged_worker_probe.json. The merged artifact’s top-level worker_probe_path, next to merged_output_path, names that file. It is written only when the merged artifact has an engine block, and rank artifacts do not carry it. The probe neither reads nor rewrites the rank and merged artifacts, which keep their startup engine block. The sidecar is written atomically after the merged artifact. When it is missing, no post-run worker evidence was recorded, for example because the probe could not write it. Dynamo deletes a sidecar left by an earlier run before it writes the merged artifact, and the sidecar’s run_id matches the merged artifact’s. The get_perf_metrics endpoint serves the merged document with the merged probe result applied to its engine block.

The following example shows selected fields from the sidecar of a two-rank run whose RPC returned only DP rank 0’s reply. Each probe result’s worker_probe is described in Worker Probe Coverage.

{
  "schema": "dynamo.fpm.benchmark_worker_probe",
  "schema_version": 1,
  "run_id": "3f6a0c1e9b2d4e7f8a5b6c7d8e9f0a1b",
  "merged_output_path": "/tmp/benchmark_results_merged.json",
  "rank_files": [
    "/tmp/benchmark_results.json",
    "/tmp/benchmark_results_dp1.json"
  ],
  "merged": {
    "resolved": {
      "tp_rank": 0,
      "pp_rank": 0,
      "pcp_rank": 0,
      "worker_rank": 0,
      "dp_rank": 0,
      "attention_backends": {
        "model.layers.0.self_attn": "FLASH_ATTN_MLA"
      },
      "mla_prefill_backend": "FlashAttnPrefillBackend",
      "cudagraph_mode_resolved": "FULL",
      "cudagraph_capture_sizes_resolved": [1, 2, 4, 8, 16, 32],
      "capture_errors": {}
    },
    "resolved_scope": "representative_worker",
    "resolution": "worker_probe_partial",
    "attention": {
      "backend_resolved": "FLASH_ATTN_MLA",
      "mla_prefill_backend_resolved": "FlashAttnPrefillBackend",
      "resolution": "worker_probe_partial"
    }
  },
  "ranks": {
    "0": {
      "rank_file": "/tmp/benchmark_results.json",
      "resolved_scope": "representative_worker",
      "resolution": "worker_probe"
    },
    "1": {
      "rank_file": "/tmp/benchmark_results_dp1.json",
      "resolved": null,
      "resolved_scope": null,
      "resolution": "worker_probe_unobserved"
    }
  }
}

Each probe result has these fields:

A rank’s result uses only replies matching its own DP rank; its resolved stays null if that rank was not observed. The TP, PP, and PCP sizes a rank’s result expects come from the merged engine block, which every DP rank covered by the merged artifact shares; a reply whose worker ranks contradict those sizes gets an issues entry and is not counted as observed. The merged result can use one observed worker as a representative, but this does not establish that every worker agrees. Use the coverage fields below to interpret the observation.

Worker Probe Coverage

Each probe result’s worker_probe preserves every returned reply, including duplicates and malformed replies, rather than selecting only the first result. A worker reply includes DP, tensor-parallel (TP), pipeline-parallel (PP), prefill-context-parallel (PCP), and worker ranks; per-layer attention_backends; mla_prefill_backend; resolved graph mode and capture sizes; and any per-layer capture_errors.

The engine client’s returned reply set can cover fewer DP engines than the RPC broadcasts to. Coverage follows the replies actually returned. For example, a four-rank run with TP, PCP, and PP sizes all set to 1 can return only DP rank 0. The merged result then reports missing DP ranks [1, 2, 3] and complete: false. Rank 0’s result can have complete local coverage; the other ranks’ results preserve the returned replies but have no local representative. Missing replies do not establish agreement or disagreement.

Each probe result’s resolution records the outcome; the artifacts keep pending_worker_probe:

attention.backend_resolved is the single backend name across valid, in-scope layer observations, or null when there are none or multiple names. attention.mla_prefill_backend_resolved similarly requires agreement among those observations. attention.resolution is worker_probe_mixed for mixed layer backend names; otherwise it follows the engine’s resolution. These summaries do not extend coverage beyond the recorded replies. Probe or sidecar-write failures leave the collected measurements usable.

Related

• Observe a Local Deployment
• Observability Architecture
• SGLang Observability