> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Environment Variables This page catalogs the environment variables that govern Dynamo observability. Set them on every frontend, router, or worker that should emit a signal. For local procedures, see [Observe a Local Deployment](/dynamo/dev/cli/operations/observability) and [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#check-deployment-health). The canonical list of variable names lives in [`lib/runtime/src/config/environment_names.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/config/environment_names.rs), which groups them by area (logging, OTLP, system server, canary, request trace). The logging and OTLP variables are read in [`lib/runtime/src/logging.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/logging.rs) `setup_logging()`; the `DYN_SYSTEM_*` and health variables are read by the system-status server. The backend log-level variables (`VLLM_LOGGING_LEVEL`, `TLLM_LOG_LEVEL`) are read by the respective engines, not the Dynamo runtime. Every variable on this page is set the same way whether you run locally or on Kubernetes — the same name, only the value or convention differs. A few fields note an **Operator preset**: the value the Dynamo operator injects on Kubernetes, which your own configuration overrides. Kubernetes-only *metrics enablement* is controlled by CRD annotations and Helm values rather than environment variables — see [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics#kubernetes-enablement-knobs). > **Info** > > Prometheus metric families in Dynamo are registered lazily: each label set is created the first time it fires, so a freshly-started process shows empty metric families until the first relevant request. An idle cluster does not mean scraping is broken. ## Setting these variables Every Dynamo process reads its own environment at startup, so set these on each process you want to emit signals — the frontend, the router, and each worker. The tabs below cover the two ways to do that; everything inside applies to whichever environment you pick. #### Local Export the variables in the shell before launching each process, then start the frontend and one or more workers: ```bash export DYN_SYSTEM_PORT=8081 export DYN_LOGGING_CONSOLE_FORMAT=jsonl export DYN_LOG=info # Frontend (serves dynamo_frontend_* on DYN_HTTP_PORT, default 8000) python -m dynamo.frontend & # vLLM worker (serves dynamo_component_* on DYN_SYSTEM_PORT) python -m dynamo.vllm --model Qwen/Qwen3-0.6B & ``` Use `python -m dynamo.sglang` or `python -m dynamo.trtllm` for the other backends, and `python -m dynamo.router` for a standalone router. #### Kubernetes Set the variables on the [`DynamoGraphDeployment`](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment) (DGD) — at graph level in `spec.envs` (prepended to every component) or per-component under `services..extraPodSpec.mainContainer.env`. A component-level entry overrides a graph-level one of the same name. ```yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: name: vllm-agg-tracing spec: envs: # applied to every component - name: DYN_LOGGING_CONSOLE_FORMAT value: "jsonl" - name: OTEL_EXPORT_ENABLED value: "true" - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT value: "http://tempo.observability.svc.cluster.local:4317" services: Frontend: componentType: frontend extraPodSpec: mainContainer: env: # applied to this component only - name: OTEL_SERVICE_NAME value: "dynamo-frontend" ``` > **Note** > > A [`DynamoGraphDeploymentRequest`](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment-request) (DGDR) has no environment-variable fields of its own. To set these under a DGDR, put them in the embedded DGD it produces via `spec.overrides.dgd`. You never set these on the operator deployment itself — the operator only writes them into the workload pods it creates; it never reads them from its own environment. ## System and metrics **`DYN_SYSTEM_PORT`** `integer` — default: -1 (disabled) Port for the backend component's system-status server, which serves `/metrics` and the health endpoints. Disabled by default; local examples use `8081`. Must be set explicitly for backend workers and the standalone router to expose metrics. **Operator preset:** `9090` on worker, prefill, decode, and planner pods. --- **`DYN_HTTP_PORT`** `integer` — default: 8000 Frontend HTTP port, where `dynamo_frontend_*` metrics are served at `/metrics`. Also configurable via the `--http-port` flag. **Operator preset:** `8000` on frontend pods. --- **`NIXL_TELEMETRY_ENABLE`** `string` — default: n Enables [NIXL](https://github.com/ai-dynamo/nixl) telemetry. NIXL metrics track KV cache and embedding data transfers and are populated only during disaggregated serving or multimodal embedding transfers. This variable alone does not produce a scrapeable endpoint: set `NIXL_TELEMETRY_EXPORTER` as well. `NIXL_TELEMETRY_PROMETHEUS_PORT` then chooses the port, and while it is optional, leaving it unset lands the exporter on a port the operator already uses. Both are described below. Allowed values: y 1 yes on true enable n 0 no off false disable NIXL compares the value case-insensitively but does not trim whitespace, so `Y` and `TRUE` are accepted while ` y` and `y ` are errors. Any other value is also an error. For SGLang operator port declarations, use `y` (case-insensitive). Other literal truthy values retain their existing port declarations to avoid rolling unchanged workloads on operator upgrade. Changing the custom resource value to `y` opts in to the additional rank ports and rolls the workload. Runtime telemetry still accepts all the truthy values listed above. **Operator preset:** `n` on worker, prefill, and decode pods. --- **`NIXL_TELEMETRY_EXPORTER`** `string` NIXL telemetry exporter to use, e.g. `prometheus`. Required for a Prometheus endpoint, with the exact lowercase value `prometheus`. NIXL has no default exporter: when this variable is unset, telemetry is collected in-process without opening a Prometheus port. Other valid exporters, such as `doca`, use their own configuration; an exporter that NIXL cannot load makes agent construction fail. **Operator preset:** `prometheus` on worker, prefill, and decode pods. --- **`NIXL_TELEMETRY_PROMETHEUS_PORT`** `integer` — default: 9090 (NIXL fallback) Port for NIXL's Prometheus exporter — a separate port from the Dynamo metrics port. Each NIXL agent binds its own exporter, so every agent needs a port of its own; two agents pointed at the same port leave the second exporter unable to bind. Give each worker a non-overlapping value when workers share a network namespace. Because each value is the base of an eight-port range (see below), `19090` for the prefill worker leaves `19098` as the next free base for the decode worker. Within a single worker, treat this as the **base** of a range rather than one port. SGLang runs one scheduler process per node-local GPU and each builds its own NIXL agent, so Dynamo gives node-local rank *i* the port `base + i` and the operator declares the whole range on the pod as `nixl`, `nixl-1`, … so Prometheus scrapes every rank. Leave room for as many consecutive ports as the pod has GPUs (up to 8). Optional: with the exporter selected, NIXL binds a listener whether or not this is set. `19090` is Dynamo's convention rather than a NIXL default, and with this variable unset the exporter falls back to its own built-in `9090` — which is the `DYN_SYSTEM_PORT` operator preset, so set it explicitly to keep the two apart. A malformed or out-of-range value is not a fallback. NIXL 1.3.2 logs the exporter creation failure and continues without an exporter; NIXL 1.4.0 raises an error during agent construction. Dynamo's own port validation rejects these values before worker startup, independently of that version-specific NIXL behavior. NIXL accepts `0` as an ephemeral port, but Dynamo rejects it because that port cannot be declared or configured as a scrape target. Set a literal port between 1 and 65535, in decimal or hexadecimal (for example, `0x4a92` for `19090`). When reserving SGLang Prometheus rank ports, the operator rejects a value taken from `valueFrom`: it cannot resolve that value to declare the range as container ports. **Operator preset:** `19090` on worker, prefill, and decode pods. --- ## OpenTelemetry (traces and logs) **`OTEL_EXPORT_ENABLED`** `boolean` — default: false Master switch for OTLP export. Gates both traces and runtime logs. Request trace records require the separate `otel` request-trace sink. --- **`OTEL_EXPORTER_OTLP_ENDPOINT`** `string` Generic endpoint for traces and logs. For `grpc`, Dynamo uses it as-is. For `http/protobuf`, Dynamo appends `/v1/traces` or `/v1/logs`. When unset, each signal uses its protocol default. --- **`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`** `string` Trace-only endpoint, used as-is. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317` for `grpc` or `http://localhost:4318/v1/traces` for `http/protobuf`. --- **`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`** `string` Log-only endpoint, used as-is. Falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`, then `http://localhost:4317` for `grpc` or `http://localhost:4318/v1/logs` for `http/protobuf`. It does not fall back to the traces endpoint. --- **`OTEL_EXPORTER_OTLP_PROTOCOL`** `string` — default: grpc Default OTLP transport for traces and logs. Allowed values: grpc http/protobuf --- **`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`** `string` Trace-only protocol override. Falls back to `OTEL_EXPORTER_OTLP_PROTOCOL`. --- **`OTEL_EXPORTER_OTLP_LOGS_PROTOCOL`** `string` Log-only protocol override. Falls back to `OTEL_EXPORTER_OTLP_PROTOCOL`. --- **`OTEL_TRACES_SAMPLE_RATIO`** `number` Head-sampling ratio in `[0.0, 1.0]`. Unset exports every trace. For example, `0.01` retains approximately one percent of traces. --- **`OTEL_SERVICE_NAME`** `string` — default: dynamo Service name attached to exported telemetry. Use a per-component value such as `dynamo-frontend` to distinguish services. --- ## Logging **`DYN_LOGGING_CONSOLE_FORMAT`** `string` — default: readable Console output format. Supported values are `readable` and `jsonl`; surrounding whitespace is ignored. When unset or blank, Dynamo falls back to `DYN_LOGGING_JSONL` for backward compatibility. This setting does not control OTLP export. Allowed values: readable jsonl --- **`DYN_LOGGING_JSONL`** `boolean` — default: false Legacy JSONL switch used when `DYN_LOGGING_CONSOLE_FORMAT` is unset or blank. It also remains a local trace-context enablement signal for backward compatibility when the new setting explicitly selects readable output. --- **`DYN_LOGGING_SPAN_EVENTS`** `boolean` — default: false Emit span entry/close events (`SPAN_FIRST_ENTRY`, `SPAN_CLOSED` messages). --- **`DYN_LOG`** `string` — default: info Log level, optionally per target: `,=,...`. Example: `info,dynamo_runtime::system_status_server:trace`. --- **`DYN_LOG_USE_LOCAL_TZ`** `boolean` — default: false Use the local timezone for log timestamps. Default is UTC. --- **`DYN_LOGGING_CONFIG_PATH`** `string` Path to a custom TOML logging configuration. Unset by default. --- **`VLLM_LOGGING_LEVEL`** `string` — default: INFO vLLM backend log level. Completely independent of `DYN_LOG`. Allowed values: DEBUG INFO WARNING ERROR CRITICAL --- **`TLLM_LOG_LEVEL`** `string` — default: INFO TensorRT-LLM backend log level. Independent of `DYN_LOG` and read **once at import time** — it must be set before the process starts. Allowed values: TRACE DEBUG INFO WARNING ERROR INTERNAL\_ERROR --- **`DYN_SKIP_SGLANG_LOG_FORMATTING`** `boolean` — default: false Disable Dynamo's SGLang log configuration so you can manage the SGLang engine's logging independently (e.g. via the worker's `--log-level` flag). --- ## Health checks **`DYN_HTTP_SVC_HEALTH_PATH`** `string` — default: /health Path of the frontend health endpoint. --- **`DYN_HTTP_SVC_LIVE_PATH`** `string` — default: /live Path of the frontend liveness endpoint. --- **`DYN_SYSTEM_STARTING_HEALTH_STATUS`** `string` — default: notready Initial health status a component reports before its required endpoints are served. Allowed values: ready notready --- **`DYN_SYSTEM_HEALTH_PATH`** `string` — default: /health Path of the health endpoint on the system-status server. --- **`DYN_SYSTEM_LIVE_PATH`** `string` — default: /live Path of the liveness endpoint on the system-status server. --- **`DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS`** `string` Deprecated compatibility variable. The current runtime logs a warning and does not use this value to select endpoints for system health. **Operator preset:** `["generate"]` on worker, prefill, and decode pods. --- **`DYN_HEALTH_CHECK_ENABLED`** `boolean` — default: false Enables canary health checks, which actively probe worker endpoints during idle periods. Defaults to `false`. The shadow-engine failover path may re-enable it per engine. (The health-check *guide* describes canary checks as "automatically enabled" in Kubernetes — that reflects the intended failover behavior, not the base value the operator injects.) **Operator preset:** `false` on worker, prefill, and decode pods. --- **`DYN_CANARY_WAIT_TIME`** `integer` — default: 10 Seconds of endpoint idle time before a canary health check is sent. Lower values check more frequently. --- **`DYN_HEALTH_CHECK_REQUEST_TIMEOUT`** `integer` — default: 3 Maximum seconds to wait for a canary health-check response before marking the endpoint unhealthy. --- **`DYN_HEALTH_CHECK_PAYLOAD`** `string` Optional canary payload override for unified backends. Accepts a JSON object or `@/path/to/payload.json` and takes precedence over the backend's default payload. --- ## Forward Pass Metrics tracing **`DYN_FPM_TRACE`** `boolean` Enables Forward Pass Metrics persistence. Equivalent to the worker `--fpm-trace` switch. --- **`DYN_FPM_OUTPUT_PATH`** `string` — default: /tmp/dynamo-fpm Output prefix for `...jsonl.gz` files. --- **`DYN_FPM_MODE`** `string` — default: sampled Capture mode: `sampled` writes the latest changed value per worker and data-parallel rank; `full` writes every valid payload. --- **`DYN_FPM_SAMPLE_INTERVAL_MS`** `integer` — default: 5000 Sampling interval used by `sampled` mode. --- **`DYN_FPM_JSONL_GZ_ROLL_BYTES`** `integer` — default: 268435456 Segment roll threshold in uncompressed JSONL bytes. --- **`DYN_FPM_MAX_SEGMENTS`** `integer` — default: 4 Number of segments retained independently for each producer. --- See [Forward Pass Metrics Trace Reference](/dynamo/dev/reference/observability/forward-pass-metrics-traces) for topology support and file semantics. ## FPM self-benchmark collection These variables tune the vLLM worker's self-benchmark collector (`--benchmark-mode`), which measures whole-model forward-pass latencies for the FPM performance database. They have no effect on serving. **`DYN_BENCH_KV_WARMUP`** `string` — default: on KV warm-up master switch. When on, eligible decode benchmark points read attention KV produced by real prefill. Set `off` to force the legacy synthetic-KV path. Models whose decode timing depends on KV content (MoE models, or models with recurrent-state layers) use native prefill when every cache group is `FullAttentionSpec`, a finite `SlidingWindowSpec`, or a Mamba/KDA state group, and at least one group is a finite window or a state group: each point prefills its own requests to the exact admission context, and the same requests continue into decode, so window history and recurrent state are computed by the model instead of borrowed from a chain at another depth. This covers Inkling's convolution cache and hybrid models such as GLM-5.3-Flash. It does not require expert parallelism or prefix caching, and `--benchmark-hybrid-live-state` has no effect when it applies. The cache-group check accepts subclasses, so MLA attention (`MLAAttentionSpec`), sliding-window MLA (`SlidingWindowMLASpec`), and k-pool tail (`KpoolTailSpec`) groups pass the per-group check; circular-buffer (`CircularBufferSpec`) groups do not. Dense attention-only models keep synthetic KV, which is correct by construction for them. The artifact records `kvwarm.initialization_strategy = "native_exact_context"`. Each decode execution, including eager warm-up replicas and repeated coordinates, prefills its own requests and shares no prefix with other executions, so the untimed prefill per sweep is about the sum of `total_kv_read_tokens` over the decode executions. Other eligible configurations use shared prefill chains and require prefix caching. By default, chain eligibility requires a MoE model with expert parallelism and no recurrent-state layers. With `DYN_BENCHMARK_RANDOMIZE_KDA_STATE=true`, hybrid MoE models keep the shared-chain random-state strategy, also without expert parallelism: their attention KV is real and their recurrent states remain synthetic. This option disables the native exact-context strategy, even for layouts that qualify for it. Both strategies require a loadable tokenizer and enough seeding dataset content for the model length. Configurations that do not meet these requirements skip warm-up and record the reason. A point whose warm-up footprint exceeds the cache pool is recorded in `kvwarm.capacity_fallbacks` and measured on synthetic KV (`fake_fallback`), or dropped from the grid under attention data parallelism. Under attention data parallelism the decision is group-wide: if any rank is ineligible, every rank uses synthetic KV; in an eligible group, a generated point that the plan cannot cover is dropped from the grid and one whose warm-up stage fails on any rank is skipped (`stage_failed_under_attention_dp`), while an explicit benchmark point (`DYN_BENCHMARK_POINTS_FILE`) in either case raises an error instead. Outside attention data parallelism, points the plan cannot cover and points whose warm-up stage fails fall back to synthetic KV (`fake_fallback`). Failed initialization never marks synthetic state as `real_kv`. --- **`DYN_BENCHMARK_RANDOMIZE_KDA_STATE`** `boolean` — default: false **Experimental.** Equivalent to `--benchmark-randomize-kda-state`. Requires `--benchmark-mode decode` or `agg`, recurrent cache groups, and the standard vLLM GPU worker with either model runner V1 or V2; custom workers and GMS are unsupported. Applies to every node, including headless workers. Initializes each decode measurement request's private KDA/Mamba state with deterministic uniform values in `[-0.01, 0.01]`, in the cache tensors' existing dtypes. The generator is seeded by request ID, layer, rank, and tensor index without changing the model's RNG stream. Initialization completes in the discarded admission step; steady decode steps update these states normally. Initialization is disabled before serving starts. With `DYN_BENCH_KV_WARMUP=on`, hybrid MoE models can use real attention KV from prefill chains even without expert parallelism. Their recurrent states remain synthetic; the source chain's states are never borrowed or modified. Existing dataset, prefix-cache, and capacity requirements still apply. Without a usable chain, attention KV follows the existing synthetic fallback. Artifacts distinguish `real_attention_kv_random_kda` from `fake_attention_kv_random_kda` and record the distribution in `recurrent_state`; a run the warm-up gate rejects labels its decode rows `skip:`, not `fake_attention_kv_random_kda`. These values are performance-test inputs, not valid context history or a guarantee of real-workload performance parity. --- **`DYN_BENCH_KV_WARMUP_DATASET`** `string` Seeding-text dataset: a local path or an `http(s)` URL. Defaults to the ShareGPT V3 conversation dump; the default download is verified against a digest pinned in code. --- **`DYN_BENCH_KV_WARMUP_CACHE_DIR`** `string` Cache directory for a downloaded dataset. Defaults to `fpm_datasets/` next to `HF_HOME`, falling back to `/tmp/fpm_datasets`. --- **`DYN_BENCH_KV_WARMUP_SHA256`** `string` Expected sha256 of the dataset file. Required to pin a custom `DYN_BENCH_KV_WARMUP_DATASET` download; overrides the built-in digest for the default dataset. --- **`DYN_BENCH_GIANT_KV_THRESHOLD`** `integer` — default: 1000000 Total-KV-read threshold (tokens) at or above which fake-injected decode points use `DYN_BENCH_GIANT_KV_REPEATS`. Real-KV warm-up points use that count regardless of this threshold, subject to model-length headroom. Set `0` to apply it to every point. --- **`DYN_BENCH_GIANT_KV_REPEATS`** `integer` — default: 3 Requested steady-step count for real-KV points and for fake-injected points at or above the threshold; the recorded latency is the median. These adjacent steps share one preparation and are not independent benchmark repetitions. The artifact retains their individual timings in `benchmark_measurement.raw_fpms`. The count is capped by model-length headroom, using the negotiated limit under attention data parallelism, so points may have a single sample. Native warm-up reserves the capped count; shared prefill chains reserve the configured count. Fake-injected points also fall back to one sample when cache capacity cannot fit the repeats. --- **`DYN_BENCH_CONTENT_SEED`** `string` — default: 0 Seed for benchmark prompt construction. The seed is combined with the point's coordinates, data-parallel rank, and request slot, independently of grid membership, benchmark point id, or earlier requests. Keep it fixed when comparing full-grid and subset runs; set the same value on every rank. Changing it creates a different content sample and is recorded in `measurement_protocol.content_seed`. Matching this seed does not establish identical inputs by itself. Compare the recorded per-rank prompt hashes and preparation metadata, and keep the model, tokenizer, dataset, content mode, and pool tag fixed. Prompt hashes do not cover sampled continuation tokens, KV-cache or recurrent-state tensors, or prior execution history. --- **`DYN_BENCH_PREFILL_CONTENT`** `string` Content source for synthetic benchmark prompts: unset keeps salt-seeded random token ids; `sharegpt` draws deterministic windows from a real-text token pool; `sharegpt_chain` reuses the KV warm-up chain texts so seeded and measured requests share one construction. --- **`DYN_BENCH_PREFILL_REAL_SEED`** `string` — default: off Real-KV seeding for prefill points that read past KV. Accepts `on`, `1`, or `true` (case-insensitive); anything else keeps the synthetic prefix blocks that are registered in the prefix cache but never computed. When on, the seeded prefix is first computed by a real, unbooked prefill pass (staging; slots already deep enough are not re-run), an unbooked same-shape warm shot follows, and only then does the timed request hit the real KV in the prefix cache. The expected hit length is validated before admission; on a miss the chain is re-staged once and the point is skipped on a second miss (`real_seed_injection_failed`, `real_seed_warm_injection_failed`, `real_seed_cache_validation_failed`). All ranks of an attention-DP group must agree on the setting (it is part of the grid-invariants digest). Dense-attention models measure the same either way; sparse-attention and hybrid-KV models need it for deep past-KV points. --- **`DYN_BENCH_POOL_TAG`** `string` Optional tag mixed into the real-text pool draw so repeated collections sample different content windows under otherwise identical settings. --- Every result and skipped-point entry in the artifact carries a `kv_seed_regime` field plus `kvwarm` and `synthetic_prompts` metadata blocks, so downstream consumers can filter by seeding provenance. Decode rows carry `real_kv`, `fake_fallback`, `legacy` (warm-up switched off), `skip:`, or `unstamped` (decode point that never reached injection). `fake_fallback` occurs only outside attention data parallelism: the configuration was eligible, but this point fell back to synthetic KV because of cache capacity, a failed warm-up stage, or a warm-up plan that does not cover it. `skip:` means the warm-up gate rejected the whole configuration, so every decode point used synthetic KV; `` says why, for example `skip:dense_model_content_insensitive` (synthetic KV is correct by construction) or `skip:dataset_empty`. Under attention data parallelism, ranks whose own check passed record `skip:peer_ineligible`. Prefill rows that read past KV carry `real_prefix` (real-KV seeding on) or `fake_prefix` (synthetic prefix blocks); other prefill rows are `not_applicable`. See [Measurement Protocol in the Self-Benchmark Artifact](/dynamo/dev/reference/observability/forward-pass-metrics-traces#measurement-protocol-in-the-self-benchmark-artifact) for input hashes, warmup completion records, per-sample timing intervals, and observed CUDA graph dispatch. See [Engine Provenance in the Self-Benchmark Artifact](/dynamo/dev/reference/observability/forward-pass-metrics-traces#engine-provenance-in-the-self-benchmark-artifact) for the optional `engine` block and the worker probe sidecar that records resolved worker configurations and probe coverage. ## Request tracing **`DYN_REQUEST_TRACE`** `boolean` Master switch. When enabled without an explicit record selection, emits `request_end,tool`. --- **`DYN_REQUEST_TRACE_RECORDS`** `string` — default: request\_end,tool Comma-separated record types: `request_end`, `request_payload`, and `tool`. --- **`DYN_REQUEST_TRACE_SINKS`** `string` — default: file Comma-separated sinks: `file`, `stderr`, `nats`, `otel`, and `s3`. --- **`DYN_REQUEST_TRACE_FILE_PATH`** `string` — default: /tmp/dynamo-request-trace Literal JSONL path or gzip segment prefix, depending on `DYN_REQUEST_TRACE_FILE_FORMAT`. --- **`DYN_REQUEST_TRACE_FILE_FORMAT`** `string` — default: jsonl\_gz File encoding: `jsonl` or rotating `jsonl_gz`. --- **`DYN_REQUEST_TRACE_CAPACITY`** `integer` — default: 1024 Best-effort in-process broadcast capacity. --- **`DYN_REQUEST_TRACE_NATS_SUBJECT`** `string` — default: dynamo.request\_trace.v1 Subject used by the `nats` sink. --- **`DYN_REQUEST_TRACE_OTEL_MAX_PAYLOAD_BYTES`** `integer` — default: 4194304 Maximum serialized OTLP payload attribute size. Oversized payload rows emit an incomplete marker. --- **`DYN_REQUEST_TRACE_FILE_BUFFER_BYTES`** `integer` — default: 1048576 File batching threshold in bytes. --- **`DYN_REQUEST_TRACE_FILE_FLUSH_INTERVAL_MS`** `integer` — default: 1000 Periodic file flush interval in milliseconds. --- **`DYN_REQUEST_TRACE_FILE_ROLL_BYTES`** `integer` — default: 268435456 Gzip roll threshold in uncompressed bytes. --- **`DYN_REQUEST_TRACE_FILE_ROLL_LINES`** `integer` Optional gzip roll threshold in records. --- **`DYN_REQUEST_TRACE_S3_BUCKET`** `string` Destination bucket for the `s3` sink. Required when `DYN_REQUEST_TRACE_SINKS` includes `s3`. When it is unset, the `s3` sink fails to initialize: Dynamo logs a warning carrying the underlying error, starts no sink from that `DYN_REQUEST_TRACE_SINKS` list, and keeps serving. Startup does not fail. See [Sink Behavior](/dynamo/dev/reference/observability/request-traces#sink-behavior). --- **`DYN_REQUEST_TRACE_S3_REGION`** `string` Region override for the `s3` sink. When unset, the client uses `AWS_REGION`, then `AWS_DEFAULT_REGION`, then `us-east-1`. --- **`DYN_REQUEST_TRACE_S3_PREFIX`** `string` Object key prefix for the `s3` sink. When unset, records land at the bucket root. Keys are `{prefix}/{yyyy}/{mm}/{dd}/{host}-{HHMMSS}-{run_id}-{seq}.jsonl.gz`. --- **`DYN_REQUEST_TRACE_S3_ROLL_UNCOMPRESSED_BYTES`** `integer` — default: 67108864 Batch roll threshold for the `s3` sink in uncompressed bytes. When the pending batch reaches this size it is gzipped and uploaded as one object. --- **`DYN_REQUEST_TRACE_S3_FLUSH_INTERVAL_MS`** `integer` — default: 10000 Periodic flush interval for the `s3` sink in milliseconds. A partial batch is uploaded when this elapses so low-volume traces still reach S3. --- **`DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT`** `string` Optional ZMQ PULL bind address for harness tool events. Configure it on only one process. --- **`DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_TOPIC`** `string` — default: agent-tool-events First-frame ZMQ topic filter. --- **`DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST`** `string` Comma- or whitespace-separated allowlist of HTTP header names captured on `request_payload` rows. Values are not redacted. --- See [Request Trace Reference](/dynamo/dev/reference/observability/request-traces) for compatibility aliases, sink behavior, and record semantics. ## Related * [Metrics Catalog](/dynamo/dev/reference/observability/metrics-catalog) — the `dynamo_*` metrics these variables expose. * [Metric Labels](/dynamo/dev/reference/observability/metric-labels) — the dimensions attached to those metrics. * [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) — Kubernetes-only metrics enablement knobs. * [Local Resource Monitor (Local)](/dynamo/dev/reference/observability/local-resource-monitor-local) — host-side per-process exporter.