Metrics Catalog

Dynamo exposes metrics in Prometheus exposition format at the /metrics HTTP endpoint. All Dynamo-generated metrics use the dynamo_* prefix and carry the hierarchy labels documented in Metric Labels. This page is the field catalog; for setup and dashboards see the Metrics guide.

Metric types

The type on each field below is a Prometheus metric type:

• counter — a cumulative value that only increases (or resets to zero on restart), e.g. total requests.
• gauge — a single value that can go up or down, e.g. in-flight requests.
• histogram — samples observations into configurable buckets and exposes _bucket, _sum, and _count series, e.g. request duration. Labeled histograms/counters/gauges register a metric family, not one series. The frontend latency and sequence-length histograms take their bucket ranges from environment variables — see Histogram buckets.

Where each family is exposed

Backend workers expose dynamo_component_* only when DYN_SYSTEM_PORT is set (disabled by default; the Kubernetes operator typically sets 9090, local examples use 8081). See Environment Variables.

Engine pass-through metrics (vllm:*, sglang:*, trtllm_*) are emitted by the backend engines themselves and are cataloged separately — see Metrics Comparison. NIXL transfer metrics are exposed on their own port and owned by the upstream NIXL project.

Frontend metrics

Emitted by the HTTP frontend at /metrics on port 8000 by default. Most carry a model label. For endpoints that canonicalize aliases, that label holds the primary served model name — the first --served-model-name value — so a request sent to an alias is counted under the primary. A name the frontend has not registered is counted under unknown_model.

Use this PromQL expression to calculate the five-minute token reuse ratio by model:

sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m]))
/
(
  sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m]))
  +
  sum by (model) (rate(dynamo_frontend_tokenizer_cache_uncached_tokens_total[5m]))
)

The ratio is defined only when the model has an active L1 cache and observed tokens. Partial hits increment both token counters. When the cache is disabled with DYN_TOKENIZER_CACHE=0, the per-model series are never created, so they are absent from the scrape rather than reported as zero; detect that state with absent(), since a == 0 comparison never matches a missing series. Only the literal value 0 disables the cache; values such as false or off leave it enabled. With the cache enabled, the counters remain zero when encoding fails or the tokenizer has no registered special-token boundaries.

Migration

Metrics for request migration, the mechanism that continues in-flight requests on a healthy worker when the original worker fails. See Request Migration Architecture for the internals.

Cancellation and rejection

Counters for requests that end early: cancellations (the client or frontend aborts an in-flight request) and rejections (the Frontend sheds load with HTTP 529 by default when all workers are busy). The worker-side counterparts are under Component metrics.

Per-worker load and timing gauges

dynamo_frontend_worker_* gauges appear once workers register and begin serving. They live on the frontend’s local registry (not component-scoped) and do not carry dynamo_namespace or dynamo_component labels. Frontend-only — not available on the standalone router. Labeled by worker_id, dp_rank, and worker_type.

At each HTTP scrape, these six gauges include only workers in the current committed router membership that pass hard availability checks, and only their current data-parallel (DP) ranks. The timing gauges also allow dp_rank="none" when the rank is unknown, provided the worker remains available. The worker_type label describes serving attribution: an Encode worker serving Chat/Completions reports its load and timing samples as decode, while the inventory metrics below retain its encode topology role. Removed or excluded workers disappear from the output; late request completions or load updates cannot restore their exposed series.

Router Worker Inventory

These frontend-only metrics describe each Frontend’s independent view of normalized base model-card discovery and router admission. They do not report Kubernetes Pod existence, and LoRA overlays do not count as additional workers. Both metrics carry model, target_namespace, target_component, target_endpoint, and worker_type labels. The target labels identify the discovered serving endpoint.

Admission uses first-wins semantics within each Frontend’s model and target group. The first observed checksum cohort remains the incumbent; later incompatible worker IDs are excluded individually and do not remove or replace it. If the incumbent cohort disappears, the next remaining cohort can become its successor after initialization.

Hard availability uses the committed group’s client and, when a prefill or encode hop targets that group, the hop’s local availability. A matching hop that is still activating reports no available workers. If no hop targets the group, its own client availability applies. These counts describe individual worker groups; use dynamo_frontend_model_ready to check whether the frontend has a complete routable serving topology.

Each currently observed group emits all four counts, including zero values for states with no workers. When discovery withdraws a group after its last worker disappears, its count and per-worker state series disappear on the next scrape. Retired rolling-update namespaces are not retained as permanent zero-count groups. Groups never observed by that frontend also have no series. Do not aggregate these counts across Frontends as worker-global readiness; separate Frontends can choose different incumbents from the same discovered workers.

The bounded reason values describe the current admission or availability condition:

When an incumbent is still initializing and an incompatible newcomer is discovered, counts are discovered=2, available=0, pending=1, and excluded=1. The incumbent carries state="pending",reason="initializing"; only the newcomer carries state="excluded",reason="checksum_mismatch". Once the incumbent is healthy, the same group reports discovered=2, available=1, pending=0, and excluded=1. Rejection of the newcomer never makes the incumbent pending or excluded.

To detect an unexpected absence of workers, select one frontend, model, worker role, and the namespaces belonging to one deployment. The following expression alerts when the frontend is being scraped successfully but no selected group has a positive discovered-worker count. It handles both explicit zero counts and series disappearing after group withdrawal:

(up{job="<frontend-job>", instance="<frontend-instance>"} == 1)
unless on (job, instance)
(
  sum by (job, instance) (
    dynamo_frontend_router_workers{
      job="<frontend-job>",
      instance="<frontend-instance>",
      model="<model>",
      target_namespace=~"<deployment-namespace-pattern>",
      worker_type="<worker-role>",
      state="discovered"
    }
  ) > 0
)

Replace the selectors with your scrape labels and a namespace pattern that includes only the rolling versions of one deployment. Enable the rule only for deployments expected to have workers, and use an alert duration such as for: 3m to tolerate startup and brief rollout transitions. To alert when no selected worker is available for routing, use state="available" instead of state="discovered". Monitor up == 0 separately for frontend scrape failures. A raw sum(...) == 0 expression does not match when all selected group series have disappeared.

Preserve target_namespace when diagnosing an individual namespace. Do not aggregate independent deployments or different frontend instances.

Inspect workers still discovered but excluded from routing:

dynamo_frontend_router_worker_state{state="excluded"} == 1

Scope these queries to the target frontend and deployment. Evaluate each Frontend separately because discovery order and first-wins admission can produce different incumbents. A missing series does not mean zero workers; check Prometheus up separately for scrape health. These metrics reflect updates after the frontend processes discovery and routing state changes, not at the instant a worker process terminates.

Model metrics

All dynamo_frontend_model_* gauges carry a model label.

The remaining model gauges are populated from worker backend registration. When multiple workers register the same model name, only the first instance’s configuration is recorded.

Component metrics

Emitted by backend workers (python -m dynamo.vllm, python -m dynamo.sglang, etc.) at /metrics on DYN_SYSTEM_PORT. All carry the hierarchy labels (dynamo_namespace, dynamo_component, dynamo_endpoint) — see Runtime-injected labels.

The following worker-side gauges and counter appear only when a hard concurrency cap is set with --engine-request-limit; see Worker-Side Request Admission.

Specialized components expose additional families under their own prefix, e.g. dynamo_preprocessor_* for preprocessor components.

Router metrics

The router exposes metrics for routing decisions and overhead. Not every metric appears in every deployment — see Availability by configuration.

Router request metrics

dynamo_component_router_* histograms and counters for aggregate request-level statistics. On the frontend, exposed at /metrics on the HTTP port; on the standalone router, exposed on DYN_SYSTEM_PORT. Populated per-request when --router-mode kv is active; registered with zero values in non-KV modes.

KV-cache reuse telemetry

The cache reuse funnel shows how much of the incoming prompt could be reused, and how much the worker reports reusing. Each step counts tokens, not requests.

All metric names below start with dynamo_component_router_.

These counters need no enable flag.

Compare token increases over the same time window and matching KV-routing traffic. Use a long enough window: the steps are recorded at different times, and F4 depends on worker reports arriving after routing. Short windows may therefore compare different requests. Divide each step by F0 to express it as a percentage of incoming tokens.

vLLM, SGLang, and TensorRT-LLM workers report F4 on the final response chunk of each aggregated or prefill attempt, as engine_data.kv_cache_hit with prompt_tokens and reused_tokens. reused_tokens is the engine’s per-request cached-token count, so it includes hits on host and storage tiers and KV connectors, the same tiers F2 and F3 count.

• Disaggregated decode attempts do not report, because each engine counts the KV transferred from the prefill worker as cached; read worker reuse from the prefill phase. The exception is a decode attempt that prefills locally under conditional disaggregation, which reports under phase="decode".
• Attempts take their phase from the frontend’s request tracker. A standalone router (python -m dynamo.router) does not set one, so all of its attempts are labeled aggregated.
• The router counts a report only when its prompt_tokens equals the routed prompt length. Engines count multimodal prompts after placeholder expansion, so a multimodal report counts only when the router also routed on expanded tokens; TensorRT-LLM does not report multimodal requests. F2 and F3 count expanded routing tokens while F0 counts the unexpanded prompt, so multimodal ratios to F0 can exceed 100%.
• With n > 1, vLLM and SGLang do not report. Later samples reuse the prompt blocks earlier samples just cached, vLLM keeps the first buffered sample’s count when it merges outputs, and SGLang caches the prompt before running the samples.
• Engines recompute the last prompt token, so a full-prefix hit reports at least one token less than F3, and up to one block less for vLLM.
• Preemption, partial-block reuse in TensorRT-LLM, and KV loads that fail under a recompute policy can also make F4 differ from F3 when the router’s estimate is accurate.

Per-request routing overhead

dynamo_router_overhead_* histograms (milliseconds) track time spent in each phase of the routing decision. Registered on the frontend port with a router_id label (the frontend’s discovery instance ID). Created only when the frontend has DRT discovery enabled (--router-mode kv); absent in non-KV modes and on the standalone router.

Router queue metrics

The frontend registers these metrics when queueing is enabled by --router-queue-threshold or a threshold in --router-policy-config. They carry model, worker_type, and policy_class. With policy-family or cache-bucket configuration, policy_class is the resolved physical queue.

KV indexer metrics

The standalone indexer exposes the device-tier-only counter dynamo_kvrouter_kv_cache_events_applied; see Standalone KV Indexer.

Availability by configuration

Not all router metrics appear in every deployment. This matrix shows which groups are registered and populated in each configuration.

Key:

• Registered and populated — the metric appears at /metrics with real values.
• Registered, always zero — the metric appears at /metrics but is never incremented (useful for dashboards that expect it to exist).
• Not registered / Not created — the metric does not appear at /metrics at all.

Endpoint picker metrics

Emitted by the endpoint picker (EPP) when routing through the Gateway API Inference Extension, at /metrics on DYN_EPP_METRICS_PORT (default 9090). Set the port to 0 to disable the endpoint.

Requires the client to request usage accounting on streaming calls ("stream_options": {"include_usage": true}); without it the response carries no usage block and nothing is recorded.

Related

• Metric Labels — the dimensions attached to these metrics.
• Environment Variables — variables that enable and configure metric emission.
• Operator Metrics — Kubernetes operator metrics catalog.
• Metrics Comparison — engine pass-through metrics (vllm:*, sglang:*, trtllm_*).