Router Guide

Overview

The Dynamo KV Router intelligently routes requests by evaluating their computational costs across different workers. It considers both decoding costs (from active blocks) and prefill costs (from newly computed blocks), using KV cache overlap to minimize redundant computation. Optimizing the KV Router is critical for achieving maximum throughput and minimum latency in distributed inference setups. This guide helps you get started with using the Dynamo router and points to the pages that cover routing concepts, configuration, disaggregated serving, and operations in more detail.

Quick Start

The router can be deployed using Python / CLI, Kubernetes, or as a standalone component.

Python / CLI Deployment

To launch the Dynamo frontend with the KV Router:

$
python -m dynamo.frontend --router-mode kv --http-port 8000

This command:

• Launches the Dynamo frontend service with KV routing enabled
• Exposes the service on port 8000 (configurable)
• Automatically handles all backend workers registered to the Dynamo endpoint

Backend workers register themselves using the register_model API. For accurate prefix-cache state, workers must also publish KV cache events with the backend-specific event flags; otherwise the router can run in approximate mode with --no-router-kv-events.

CLI Arguments

For all available options: python -m dynamo.frontend --help

For detailed configuration options and tuning parameters, see Configuration and Tuning. For how the router models prefill and decode load in the cost function, see Routing Concepts.

Kubernetes Deployment

To enable the KV Router in Kubernetes, add the DYN_ROUTER_MODE environment variable to your frontend service:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
spec:
6
  services:
7
    Frontend:
8
      componentType: frontend
9
      replicas: 1
10
      envs:
11
        - name: DYN_ROUTER_MODE
12
          value: kv  # Enable KV Smart Router

Key Points:

• Set DYN_ROUTER_MODE=kv on the Frontend service only
• Configure worker-side KV event publishing when you want event-driven prefix-cache state
• Use --no-router-kv-events for approximate cache-state prediction when workers are not publishing events

Environment Variables

All CLI arguments can be configured via environment variables using the DYN_ prefix:

For complete K8s examples and advanced configuration, see K8s Examples and Configuration and Tuning. For A/B testing and advanced K8s setup, see the KV Router A/B Benchmarking Guide.

Standalone Router

You can also run the KV router as a standalone service (without the Dynamo frontend) for disaggregated serving (e.g., routing to prefill workers), multi-tier architectures, or any scenario requiring intelligent KV cache-aware routing decisions. See the Standalone Router component for more details.

Frontend-Embedded vs. Standalone Router

The standalone router does not include the HTTP frontend (no /v1/chat/completions endpoint). It exposes only the RouterRequestMetrics via the system status server. See the Standalone Router README.

Deployment Modes

The Dynamo router can be deployed in several configurations. The table below shows every combination and when to use it:

Routing Modes (--router-mode)

Device-Aware Weighted Routing

device-aware-weighted is designed for heterogeneous fleets where workers of different compute capability, for example CPU embedding encoders alongside GPU embedding encoders, share the same endpoint.

Workers are split into CPU and non-CPU groups. The router compares a capability-normalized load across the two groups:

normalized_load = total_inflight(group) / (instance_count(group) x throughput_weight)

The throughput weight is 1 for CPU workers and DYN_ENCODER_CUDA_TO_CPU_RATIO for non-CPU workers. The next request is routed to the group with the lower normalized load, then to the least-loaded worker inside that group.

Use DYN_ENCODER_CUDA_TO_CPU_RATIO to approximate the throughput ratio of a non-CPU worker relative to one CPU worker. The default is 8.

When only one device class is present, the policy degenerates to standard least-loaded routing.

KV Event Transport Modes (within --router-mode kv)

When using KV routing, the router needs to know what each worker has cached. There are four ways to get this information:

Aggregated vs. Disaggregated Topology

Disaggregated mode is activated automatically when prefill workers register alongside decode workers. See Disaggregated Serving for details.

More Router Docs

• Routing Concepts: Cost model, worker selection, and routing primitives
• Configuration and Tuning: Router flags, transport modes, load tracking, and metrics
• Disaggregated Serving: Prefill and decode routing setups
• Router Operations: Replicas, remote indexers, persistence, and recovery
• Router Examples: Python API usage, K8s examples, and custom routing patterns
• Router Testing: Recommended test layers for non-trivial router changes
• Standalone Indexer: Run the KV indexer as a separate service
• KV Event Replay — Dynamo vs vLLM: Gap detection and replay behavior