> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Metrics Catalog Dynamo exposes metrics in Prometheus exposition format at the `/metrics` HTTP endpoint. All Dynamo-generated metrics use the `dynamo_*` prefix and carry the hierarchy labels documented in [Metric Labels](/dynamo/dev/reference/observability/metric-labels). This page is the field catalog; for setup and dashboards see the [Metrics guide](/dynamo/dev/cli/operations/observability#view-metrics-and-dashboards). ## Metric types The `type` on each field below is a [Prometheus metric type](https://prometheus.io/docs/concepts/metric_types/): * **counter** — a cumulative value that only increases (or resets to zero on restart), e.g. total requests. * **gauge** — a single value that can go up or down, e.g. in-flight requests. * **histogram** — samples observations into configurable buckets and exposes `_bucket`, `_sum`, and `_count` series, e.g. request duration. Labeled histograms/counters/gauges register a metric *family*, not one series. The frontend latency and sequence-length histograms take their bucket ranges from environment variables — see [Histogram buckets](/dynamo/dev/reference/components/frontend-configuration#histogram-buckets). > **Info** > > Labeled metrics (`HistogramVec`, `CounterVec`, `GaugeVec`) register a metric *family*, not individual time series. For request-populated metrics, a series for a given label combination appears at `/metrics` only after the first matching request. Other families create or remove series from runtime state; their entries below document those semantics. ## Where each family is exposed | Family | Emitted by | Scrape endpoint | | -------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------- | | `dynamo_frontend_*` | HTTP frontend (`python -m dynamo.frontend`) | `/metrics` on `DYN_HTTP_PORT` (default 8000) | | `dynamo_component_*` | Backend workers, standalone router | `/metrics` on `DYN_SYSTEM_PORT` (must be set) | | `dynamo_router_overhead_*` | Frontend with KV routing | `/metrics` on `DYN_HTTP_PORT` | | `dynamo_epp_*` | Endpoint picker (Gateway API Inference Extension routing) | `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090) | | `dynamo_operator_*` | Kubernetes operator | See [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) | Backend workers expose `dynamo_component_*` only when `DYN_SYSTEM_PORT` is set (disabled by default; the Kubernetes operator typically sets `9090`, local examples use `8081`). See [Environment Variables](/dynamo/dev/reference/observability/environment-variables#system-and-metrics). Engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`) are emitted by the backend engines themselves and are cataloged separately — see [Metrics Comparison](/dynamo/dev/reference/observability/metrics-comparison). NIXL transfer metrics are exposed on their own port and owned by the [upstream NIXL project](https://github.com/ai-dynamo/nixl/blob/main/docs/telemetry.md). ## Frontend metrics Emitted by the HTTP frontend at `/metrics` on port 8000 by default. Most carry a `model` label. For endpoints that canonicalize aliases, that label holds the primary served model name — the first `--served-model-name` value — so a request sent to an alias is counted under the primary. A name the frontend has not registered is counted under `unknown_model`. **`dynamo_frontend_active_requests`** `gauge` Requests currently being handled by the frontend, from HTTP handler entry until the response stream completes. The top-level in-flight count with no stage breakdown. --- **`dynamo_frontend_stage_requests`** `gauge` Requests currently in a given frontend pipeline stage. Labeled by `stage` and `phase` — see [Stage values](/dynamo/dev/reference/observability/metric-labels#stage-values) and [Phase values](/dynamo/dev/reference/observability/metric-labels#phase-values). --- **`dynamo_frontend_inflight_requests`** `gauge` — deprecated Inflight requests. Kept for backward compatibility; prefer `dynamo_frontend_active_requests`, which has identical semantics with a clearer name. --- **`dynamo_frontend_queued_requests`** `gauge` — deprecated Requests in the HTTP processing queue. Kept for backward compatibility; the "waiting for first token" window is now the sum of `dynamo_frontend_stage_requests` across the `preprocess`, `route`, and `dispatch` stages. --- **`dynamo_frontend_disconnected_clients`** `gauge` Number of disconnected clients. --- **`dynamo_frontend_input_sequence_tokens`** `histogram` Input sequence length in tokens. --- **`dynamo_frontend_cached_tokens`** `histogram` Cached tokens (prefix cache hits) per request. --- **`dynamo_frontend_tokenizer_cache_hits_total`** `counter` L1 tokenizer prefix-cache hits across all models. One outcome is recorded per encode operation or batch item. --- **`dynamo_frontend_tokenizer_cache_misses_total`** `counter` L1 tokenizer prefix-cache misses across all models. --- **`dynamo_frontend_tokenizer_cache_cached_tokens_total`** `counter` Tokens returned from the L1 tokenizer prefix cache. Labeled by `model`. --- **`dynamo_frontend_tokenizer_cache_uncached_tokens_total`** `counter` Tokens freshly encoded after an L1 tokenizer cache lookup. Labeled by `model`. --- Use this PromQL expression to calculate the five-minute token reuse ratio by model: ```promql sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) / ( sum by (model) (rate(dynamo_frontend_tokenizer_cache_cached_tokens_total[5m])) + sum by (model) (rate(dynamo_frontend_tokenizer_cache_uncached_tokens_total[5m])) ) ``` The ratio is defined only when the model has an active L1 cache and observed tokens. Partial hits increment both token counters. When the cache is disabled with `DYN_TOKENIZER_CACHE=0`, the per-model series are never created, so they are absent from the scrape rather than reported as zero; detect that state with `absent()`, since a `== 0` comparison never matches a missing series. Only the literal value `0` disables the cache; values such as `false` or `off` leave it enabled. With the cache enabled, the counters remain zero when encoding fails or the tokenizer has no registered special-token boundaries. **`dynamo_frontend_inter_token_latency_seconds`** `histogram` Inter-token latency in seconds. --- **`dynamo_frontend_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_frontend_output_tokens_total`** `counter` Total output tokens generated. --- **`dynamo_frontend_request_duration_seconds`** `histogram` End-to-end LLM request duration in seconds. --- **`dynamo_frontend_requests_total`** `counter` Total LLM requests. --- **`dynamo_frontend_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- ### Migration Metrics for [request migration](/dynamo/dev/kubernetes/fault-tolerance/request-migration), the mechanism that continues in-flight requests on a healthy worker when the original worker fails. See [Request Migration Architecture](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-migration-architecture) for the internals. **`dynamo_frontend_model_migration_total`** `counter` Total request migrations due to worker unavailability. Labeled by `model` and [`migration_type`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels) (`new_request` for an initial connection failure, `ongoing_request` for a mid-stream disconnection). --- **`dynamo_frontend_model_migration_duration_seconds`** `histogram` Time from detecting a migratable worker failure until the replacement response stream is established, migration ends without recovery, or the request is cancelled. Labeled by `model`, [`migration_type`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels), and [`outcome`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels). Use the histogram's `_count` series to count completed migration events by outcome. The duration excludes work completed before the failure and generation after the replacement stream is established. --- **`dynamo_frontend_model_migration_max_seq_len_exceeded_total`** `counter` Total times migration was disabled for a request because its sequence length exceeded `--migration-max-seq-len`. A rising value may indicate the limit needs adjustment. Labeled by `model`. --- ### Cancellation and rejection Counters for requests that end early: [cancellations](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture) (the client or frontend aborts an in-flight request) and [rejections](/dynamo/dev/kubernetes/fault-tolerance/request-rejection) (the Frontend sheds load with HTTP 529 by default when all workers are busy). The worker-side counterparts are under [Component metrics](#component-metrics). **`dynamo_frontend_model_cancellation_total`** `counter` Total request cancellations detected by the frontend (client disconnect or stream close). Labeled by `model`, `endpoint` (the API route — `chat_completions`, `completions`, `embeddings`, …), and [`request_type`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels) (`unary` or `stream`). --- **`dynamo_frontend_model_rejection_total`** `counter` Total overload responses surfaced to the client as HTTP 529 by default. Incremented for worker-scoped `WorkerOverloaded` and pool-scoped `ResourceExhausted` errors. Labeled by `model` and `endpoint` (the API route). --- ### Per-worker load and timing gauges `dynamo_frontend_worker_*` gauges appear once workers register and begin serving. They live on the frontend's local registry (not component-scoped) and do **not** carry `dynamo_namespace` or `dynamo_component` labels. Frontend-only — not available on the standalone router. Labeled by `worker_id`, `dp_rank`, and `worker_type`. At each HTTP scrape, these six gauges include only workers in the current committed router membership that pass hard availability checks, and only their current data-parallel (DP) ranks. The timing gauges also allow `dp_rank="none"` when the rank is unknown, provided the worker remains available. The `worker_type` label describes serving attribution: an Encode worker serving Chat/Completions reports its load and timing samples as `decode`, while the inventory metrics below retain its `encode` topology role. Removed or excluded workers disappear from the output; late request completions or load updates cannot restore their exposed series. **`dynamo_frontend_worker_active_decode_blocks`** `gauge` Active KV cache decode blocks per worker. --- **`dynamo_frontend_worker_active_prefill_tokens`** `gauge` Active prefill tokens queued per worker. --- **`dynamo_frontend_worker_active_requests`** `gauge` Active requests booked per worker, split by an additional [`request_phase`](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels) label. A request counts as `prefill` from booking until its worker returns the first output token, then as `decode` until it completes. Sum both phases for the worker's total concurrency. The count is the router's view, including requests booked by peer routers when router replica sync is enabled. --- **`dynamo_frontend_worker_last_time_to_first_token_seconds`** `gauge` Last observed time to first token per worker, in seconds. --- **`dynamo_frontend_worker_last_input_sequence_tokens`** `gauge` Last observed input sequence length per worker. --- **`dynamo_frontend_worker_last_inter_token_latency_seconds`** `gauge` Last observed inter-token latency per worker, in seconds. --- ### Router Worker Inventory These frontend-only metrics describe each Frontend's independent view of normalized base model-card discovery and router admission. They do not report Kubernetes Pod existence, and LoRA overlays do not count as additional workers. Both metrics carry `model`, `target_namespace`, `target_component`, `target_endpoint`, and `worker_type` labels. The target labels identify the discovered serving endpoint. Admission uses first-wins semantics within each Frontend's model and target group. The first observed checksum cohort remains the incumbent; later incompatible worker IDs are excluded individually and do not remove or replace it. If the incumbent cohort disappears, the next remaining cohort can become its successor after initialization. **`dynamo_frontend_router_workers`** `gauge` Number of unique physical worker IDs in a model and target group, labeled by `state`. DP ranks do not increase the count. The `available`, `pending`, and `excluded` states are disjoint; `discovered` is their sum. `available` requires successfully committed worker-set membership and current hard availability; `pending` means discovery has observed the worker but initialization has not completed; `excluded` means discovered but currently unavailable for routing. --- Hard availability uses the committed group's client and, when a prefill or encode hop targets that group, the hop's local availability. A matching hop that is still activating reports no available workers. If no hop targets the group, its own client availability applies. These counts describe individual worker groups; use `dynamo_frontend_model_ready` to check whether the frontend has a complete routable serving topology. Each currently observed group emits all four counts, including zero values for states with no workers. When discovery withdraws a group after its last worker disappears, its count and per-worker state series disappear on the next scrape. Retired rolling-update namespaces are not retained as permanent zero-count groups. Groups never observed by that frontend also have no series. Do not aggregate these counts across Frontends as worker-global readiness; separate Frontends can choose different incumbents from the same discovered workers. **`dynamo_frontend_router_worker_state`** `gauge` One sample with value `1` for each discovered worker, additionally labeled by `router_worker_id`, `state`, and `reason`. The state is `available`, `pending`, or `excluded`. The sample changes with the worker's current state and disappears when discovery removes the ID. --- The bounded `reason` values describe the current admission or availability condition: | Reason | Meaning | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | `none` | Available for routing | | `initializing` | Waiting for initialization | | `checksum_mismatch` | The individual worker's Model Deployment Card checksum differs from the incumbent; emitted only for rejected worker IDs | | `materialization_failed` | Creating the serving configuration failed | | `commit_blocked` | The serving configuration could not be committed | | `invalid_config` | The discovered configuration is invalid | | `unavailable` | Absent from the current availability view; includes local inhibition and other availability changes without identifying their exact cause | When an incumbent is still initializing and an incompatible newcomer is discovered, counts are `discovered=2`, `available=0`, `pending=1`, and `excluded=1`. The incumbent carries `state="pending",reason="initializing"`; only the newcomer carries `state="excluded",reason="checksum_mismatch"`. Once the incumbent is healthy, the same group reports `discovered=2`, `available=1`, `pending=0`, and `excluded=1`. Rejection of the newcomer never makes the incumbent pending or excluded. To detect an unexpected absence of workers, select one frontend, model, worker role, and the namespaces belonging to one deployment. The following expression alerts when the frontend is being scraped successfully but no selected group has a positive discovered-worker count. It handles both explicit zero counts and series disappearing after group withdrawal: ```promql (up{job="", instance=""} == 1) unless on (job, instance) ( sum by (job, instance) ( dynamo_frontend_router_workers{ job="", instance="", model="", target_namespace=~"", worker_type="", state="discovered" } ) > 0 ) ``` Replace the selectors with your scrape labels and a namespace pattern that includes only the rolling versions of one deployment. Enable the rule only for deployments expected to have workers, and use an alert duration such as `for: 3m` to tolerate startup and brief rollout transitions. To alert when no selected worker is available for routing, use `state="available"` instead of `state="discovered"`. Monitor `up == 0` separately for frontend scrape failures. A raw `sum(...) == 0` expression does not match when all selected group series have disappeared. Preserve `target_namespace` when diagnosing an individual namespace. Do not aggregate independent deployments or different frontend instances. Inspect workers still discovered but excluded from routing: ```promql dynamo_frontend_router_worker_state{state="excluded"} == 1 ``` Scope these queries to the target frontend and deployment. Evaluate each Frontend separately because discovery order and first-wins admission can produce different incumbents. A missing series does not mean zero workers; check Prometheus `up` separately for scrape health. These metrics reflect updates after the frontend processes discovery and routing state changes, not at the instant a worker process terminates. ### Model metrics All `dynamo_frontend_model_*` gauges carry a `model` label. **`dynamo_frontend_model_ready`** `gauge` Whether the frontend can currently route at least one inference request for the model. A value of `1` means at least one complete serving topology with a live worker and serving engine is available; `0` means the model is registered but not currently routable. The frontend evaluates this gauge from its live routing catalog at scrape time, so worker registration and removal are reflected in the next scrape. Unlike request-populated metrics, the series appears when the model registers without requiring an inference request. The series is absent while the model is not registered, and the entire family is absent when no models are registered. Detect this state with `absent(dynamo_frontend_model_ready{model=""})`; a `== 0` comparison does not match an absent series. Aliases get no series of their own: a deployment served under several names emits one series, labeled with its primary name, so query an alias through its primary. `/v1/models` still lists alias names, but only the primary has a readiness series. A LoRA adapter is a distinct model rather than a second name for one, and keeps its own series. --- The remaining model gauges are populated from worker backend registration. When multiple workers register the same model name, only the first instance's configuration is recorded. **`dynamo_frontend_model_total_kv_blocks`** `gauge` Total KV blocks available for a worker serving the model. --- **`dynamo_frontend_model_max_num_seqs`** `gauge` Maximum number of sequences for a worker serving the model. --- **`dynamo_frontend_model_max_num_batched_tokens`** `gauge` Maximum number of batched tokens for a worker serving the model. --- **`dynamo_frontend_model_context_length`** `gauge` Maximum context length for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_kv_cache_block_size`** `gauge` KV cache block size for a worker serving the model. Sourced from the Model Deployment Card. --- **`dynamo_frontend_model_migration_limit`** `gauge` Request migration limit for a worker serving the model. Sourced from the Model Deployment Card. --- ## Component metrics Emitted by backend workers (`python -m dynamo.vllm`, `python -m dynamo.sglang`, etc.) at `/metrics` on `DYN_SYSTEM_PORT`. All carry the hierarchy labels (`dynamo_namespace`, `dynamo_component`, `dynamo_endpoint`) — see [Runtime-injected labels](/dynamo/dev/reference/observability/metric-labels#runtime-injected-labels). **`dynamo_component_inflight_requests`** `gauge` Requests currently being processed by the component. --- **`dynamo_component_request_bytes_total`** `counter` Total bytes received in requests. --- **`dynamo_component_request_duration_seconds`** `histogram` Request processing time in seconds. --- **`dynamo_component_requests_total`** `counter` Total requests processed. --- **`dynamo_component_errors_total`** `counter` Total errors encountered while handling a request. Labeled by `error_type` — see [Component error types](/dynamo/dev/reference/observability/metric-labels#component-error-types). --- **`dynamo_component_response_bytes_total`** `counter` Total bytes sent in responses. --- **`dynamo_component_uptime_seconds`** `gauge` DistributedRuntime uptime. Updated before each Prometheus scrape on both the frontend and the system-status server. --- **`dynamo_component_cancellation_total`** `counter` Total requests cancelled by the work handler. Carries the hierarchy labels. Records cancellation *signals* received by the worker, not whether the engine actually aborted — deduplicated so a control message plus a socket close for the same request counts once. See [Request Cancellation Architecture](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture). --- The following worker-side gauges and counter appear only when a hard concurrency cap is set with `--engine-request-limit`; see [Worker-Side Request Admission](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-rejection-architecture#worker-side-request-admission). **`dynamo_rejection_request_total`** `counter` Cumulative requests rejected because the worker was at capacity (engine in-flight limit and Dynamo queue both full). --- **`dynamo_engine_request`** `gauge` Current requests being handled by the engine. --- **`dynamo_request_queue`** `gauge` Current requests queued in Dynamo, not yet in the engine. --- Specialized components expose additional families under their own prefix, e.g. `dynamo_preprocessor_*` for preprocessor components. ## Router metrics The router exposes metrics for routing decisions and overhead. Not every metric appears in every deployment — see [Availability by configuration](#availability-by-configuration). **`dynamo_component_router_worker_registered`** `gauge` A sample with value `1` for each worker and DP rank in the scheduler's current registration and hard-availability view. Carries `router_worker_id`, `dp_rank`, and `worker_type` labels in addition to the component labels. The sample can appear during scheduler preparation before worker-set materialization commits, so it does not indicate committed model readiness. Removed or unavailable workers disappear from the output, rather than emitting `0`; late request bookkeeping cannot restore their exposed series. Prefer the [Router Worker Inventory](#router-worker-inventory) metrics for committed available-worker counts and exclusion reasons on the frontend. --- ### Router request metrics `dynamo_component_router_*` histograms and counters for aggregate request-level statistics. On the frontend, exposed at `/metrics` on the HTTP port; on the standalone router, exposed on `DYN_SYSTEM_PORT`. Populated per-request when `--router-mode kv` is active; registered with zero values in non-KV modes. **`dynamo_component_router_requests_started_total`** `counter` Requests admitted by the router scheduler. --- **`dynamo_component_router_requests_total`** `counter` Total requests processed by the router. --- **`dynamo_component_router_time_to_first_token_seconds`** `histogram` Time to first token in seconds. --- **`dynamo_component_router_inter_token_latency_seconds`** `histogram` Average inter-token latency in seconds. --- **`dynamo_component_router_input_sequence_tokens`** `histogram` Input sequence length per routing attempt, excluding query-only lookups. Labeled by `phase` and `model`. --- **`dynamo_component_router_output_sequence_tokens`** `histogram` Output sequence length in tokens. --- **`dynamo_component_router_kv_hit_rate`** `histogram` Predicted KV cache hit rate at routing time (0.0–1.0). --- **`dynamo_component_router_non_max_overlap_selections_total`** `counter` Admitted prefill scheduler selections routed to a worker with less KV cache overlap than another eligible worker. Pinned requests and equal-overlap ties are excluded. Labeled by `worker_type` (`prefill`). --- **`dynamo_component_router_overlap_blocks_lost`** `histogram` Difference in effective KV cache overlap between the highest-overlap eligible prefill worker and the selected worker for each non-max-overlap selection. Labeled by `worker_type` (`prefill`). --- ### KV-cache reuse telemetry The cache reuse funnel shows how much of the incoming prompt could be reused, and how much the worker reports reusing. Each step counts tokens, not requests. All metric names below start with `dynamo_component_router_`. | Step | Meaning | Metric suffix | | -------------------------------- | --------------------------------------------------------------------------- | --------------------------------------------- | | **F0 — Incoming tokens** | Tokens received by the router. | `input_sequence_tokens_sum` | | **F2 — Best available match** | Prefix the router estimates is cached on the best-matching eligible worker. | `kv_best_eligible_cached_prefix_tokens_total` | | **F3 — Selected worker's match** | Prefix the router estimates is cached on the worker it chooses. | `kv_selected_cached_prefix_tokens_total` | | **F4 — Reported reuse** | Tokens the worker reports reusing. Missing reports count as zero. | `kv_worker_reused_tokens_total` | These counters need no enable flag. Compare token increases over the same time window and matching KV-routing traffic. Use a long enough window: the steps are recorded at different times, and F4 depends on worker reports arriving after routing. Short windows may therefore compare different requests. Divide each step by F0 to express it as a percentage of incoming tokens. vLLM, SGLang, and TensorRT-LLM workers report F4 on the final response chunk of each aggregated or prefill attempt, as `engine_data.kv_cache_hit` with `prompt_tokens` and `reused_tokens`. `reused_tokens` is the engine's per-request cached-token count, so it includes hits on host and storage tiers and KV connectors, the same tiers F2 and F3 count. * Disaggregated decode attempts do not report, because each engine counts the KV transferred from the prefill worker as cached; read worker reuse from the `prefill` phase. The exception is a decode attempt that prefills locally under conditional disaggregation, which reports under `phase="decode"`. * Attempts take their `phase` from the frontend's request tracker. A standalone router (`python -m dynamo.router`) does not set one, so all of its attempts are labeled `aggregated`. * The router counts a report only when its `prompt_tokens` equals the routed prompt length. Engines count multimodal prompts after placeholder expansion, so a multimodal report counts only when the router also routed on expanded tokens; TensorRT-LLM does not report multimodal requests. F2 and F3 count expanded routing tokens while F0 counts the unexpanded prompt, so multimodal ratios to F0 can exceed 100%. * With `n > 1`, vLLM and SGLang do not report. Later samples reuse the prompt blocks earlier samples just cached, vLLM keeps the first buffered sample's count when it merges outputs, and SGLang caches the prompt before running the samples. * Engines recompute the last prompt token, so a full-prefix hit reports at least one token less than F3, and up to one block less for vLLM. * Preemption, partial-block reuse in TensorRT-LLM, and KV loads that fail under a recompute policy can also make F4 differ from F3 when the router's estimate is accurate. ### Per-request routing overhead `dynamo_router_overhead_*` histograms (milliseconds) track time spent in each phase of the routing decision. Registered on the frontend port with a `router_id` label (the frontend's discovery instance ID). Created only when the frontend has DRT discovery enabled (`--router-mode kv`); absent in non-KV modes and on the standalone router. **`dynamo_router_overhead_block_hashing_ms`** `histogram` Time computing block hashes. --- **`dynamo_router_overhead_indexer_find_matches_ms`** `histogram` Time in indexer `find_matches`. --- **`dynamo_router_overhead_seq_hashing_ms`** `histogram` Time computing sequence hashes. --- **`dynamo_router_overhead_scheduling_ms`** `histogram` Time in scheduler worker selection. --- **`dynamo_router_overhead_total_ms`** `histogram` Total routing overhead per request. --- ### Router queue metrics The frontend registers these metrics when queueing is enabled by `--router-queue-threshold` or a threshold in `--router-policy-config`. They carry `model`, `worker_type`, and `policy_class`. With policy-family or cache-bucket configuration, `policy_class` is the resolved physical queue. **`dynamo_frontend_router_queue_pending_requests`** `gauge` Requests pending in the router scheduler queue. --- **`dynamo_frontend_router_queue_pending_isl_tokens`** `gauge` Raw input-sequence tokens pending in the queue. --- **`dynamo_frontend_router_queue_pending_cached_tokens`** `gauge` Cached-token estimate captured when each request enters the queue. --- **`dynamo_frontend_router_queue_backpressure_total`** `counter` Queue rejections by configured limit reason. Also labeled by `reason`. --- ### KV indexer metrics **`dynamo_component_kv_cache_events_applied`** `counter` KV cache events applied to the router's radix-tree index. Includes events applied to the device tier and lower tiers such as host-pinned memory and disk. Appears when the worker-selection policy requests `WorkerInputs::CACHE`, the router consumes KV events, and workers publish them. Labeled by `status` and `event_type` — see [Metric-specific labels](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels). --- **`dynamo_component_ckf_mutation_total`** `counter` CKF block-level mutation outcomes recorded by the router indexer, labeled by `outcome` — see [Metric-specific labels](/dynamo/dev/reference/observability/metric-labels#metric-specific-labels). **Registered, always zero.** The series is created and exported, but no code path currently increments it, so it reads 0 in every deployment. Do not build alerts or dashboard panels on it. The standalone indexer exports the same counter as `dynamo_kvrouter_ckf_mutation_total`, with the same caveat. --- The standalone indexer exposes the device-tier-only counter `dynamo_kvrouter_kv_cache_events_applied`; see [Standalone KV Indexer](/dynamo/dev/knowledge-base/modular-components/router/standalone-indexer). ### Availability by configuration Not all router metrics appear in every deployment. This matrix shows which groups are **registered** and **populated** in each configuration. | Metric group | Frontend + KV (agg) | Frontend + KV (disagg) | Frontend + non-KV | Standalone router | | ------------------------------------------ | ----------------------------------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------- | --------------------------------- | | `dynamo_component_router_*` | Registered and populated | Registered and populated | Registered, **always zero** | Populated (on `DYN_SYSTEM_PORT`) | | `dynamo_router_overhead_*` | Registered and populated | Registered and populated | **Not registered** | **Not created** | | `dynamo_frontend_router_queue_*` | Registered; populated when a CLI or policy-class queue threshold is set | Registered; populated when a CLI or policy-class queue threshold is set | **Not registered** | **Not created** | | `dynamo_component_kv_cache_events_applied` | Populated when KV events received | Populated when KV events received | **Not registered** | Populated when KV events received | | `dynamo_component_ckf_mutation_total` | Registered, **always zero** | Registered, **always zero** | **Not registered** | Registered, **always zero** | | `dynamo_frontend_worker_*` | Registered and populated | Registered and populated (`worker_type` = `prefill`/`decode`) | Registered and populated (`worker_type` = `decode`) | **Not created** | **Key:** * **Registered and populated** — the metric appears at `/metrics` with real values. * **Registered, always zero** — the metric appears at `/metrics` but is never incremented (useful for dashboards that expect it to exist). * **Not registered / Not created** — the metric does not appear at `/metrics` at all. ## Endpoint picker metrics Emitted by the endpoint picker (EPP) when routing through the Gateway API Inference Extension, at `/metrics` on `DYN_EPP_METRICS_PORT` (default 9090). Set the port to `0` to disable the endpoint. **`dynamo_epp_cached_tokens`** `histogram` Prompt tokens the model server served from its KV cache per request, read from `usage.prompt_tokens_details.cached_tokens` in the response body. This is the *observed* cache hit, as opposed to the overlap the KV router estimates at selection time. Buckets match `dynamo_frontend_cached_tokens` so the two can share a dashboard. The `model` label is the model the pool serves, resolved once at startup rather than from the request body. --- Requires the client to request usage accounting on streaming calls (`"stream_options": {"include_usage": true}`); without it the response carries no usage block and nothing is recorded. ## Related * [Metric Labels](/dynamo/dev/reference/observability/metric-labels) — the dimensions attached to these metrics. * [Environment Variables](/dynamo/dev/reference/observability/environment-variables) — variables that enable and configure metric emission. * [Operator Metrics](/dynamo/dev/reference/observability/operator-metrics) — Kubernetes operator metrics catalog. * [Metrics Comparison](/dynamo/dev/reference/observability/metrics-comparison) — engine pass-through metrics (`vllm:*`, `sglang:*`, `trtllm_*`).