Router Examples

For quick start instructions, see the Router README. This document provides further examples for using the Dynamo Router, including Python API usage, Kubernetes deployments, and custom routing patterns.

Table of Contents

• Using KvRouter Python API
• K8s Examples
• Routing Patterns
• Custom Routing Example: Minimizing TTFT
• KV Event Publishing for Custom Engines
• Global Router (Hierarchical Routing)

Using KvRouter Python API

Instead of launching the KV Router via command line, you can create a KvRouter object directly in Python. This allows per-request routing configuration overrides.

Methods

The KvRouter provides the following methods:

• generate(token_ids, model, ...): Route and execute a request, returning an async stream of responses. Automatically handles worker selection, state tracking, and lifecycle management.

• best_worker(token_ids, router_config_override=None, request_id=None): Query which worker would be selected for given tokens. Returns (worker_id, dp_rank, overlap_blocks).

  • Without request_id: Query-only, doesn’t update router state
  • With request_id: Updates router state to track the request. Note: If used with request_id, you must call mark_prefill_complete() and free() at the appropriate lifecycle points to maintain accurate load tracking

• get_potential_loads(token_ids): Get detailed load information for all workers, including potential prefill tokens and active decode blocks. Returns a list of load dictionaries.

• mark_prefill_complete(request_id): Signal that a request has completed its prefill phase. Only used for manual lifecycle management when using best_worker() for manual routing instead of generate().

• free(request_id): Signal that a request has completed and its resources should be released. Only used for manual lifecycle management when using best_worker() for manual routing instead of generate().

• dump_events(): Dump all KV cache events from the router’s indexer as a JSON string. Useful for debugging and analysis.

Setup

First, launch your backend engines:

$
python -m dynamo.vllm --model meta-llama/Llama-2-7b-hf

Example Script

1
import asyncio
2
from dynamollm import DistributedRuntime, KvRouter, KvRouterConfig
3
4
async def main():
5
    # Get runtime and create endpoint
6
    runtime = DistributedRuntime.detached()
7
    namespace = runtime.namespace("dynamo")
8
    component = namespace.component("backend")
9
    endpoint = component.endpoint("generate")
10
11
    # Create KV router
12
    kv_router_config = KvRouterConfig()
13
    router = KvRouter(
14
        endpoint=endpoint,
15
        block_size=16,
16
        kv_router_config=kv_router_config
17
    )
18
19
    # Your input tokens
20
    token_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
21
22
    # Generate with per-request routing override
23
    stream = await router.generate(
24
        token_ids=token_ids,
25
        model="meta-llama/Llama-2-7b-hf",
26
        stop_conditions={
27
            "max_tokens": 20,        # Generate exactly 20 tokens
28
            "ignore_eos": True,      # Don't stop at EOS token
29
        },
30
        sampling_options={
31
            "temperature": 0.7,
32
            "top_p": 0.9,
33
        },
34
        router_config_override={
35
            "overlap_score_weight": 2.0,    # Prioritize cache hits for this request
36
            "router_temperature": 0.5,       # Add routing randomness
37
        }
38
    )
39
40
    # Collect generated tokens
41
    generated_tokens = []
42
    async for response in stream:
43
        if isinstance(response, dict) and "token_ids" in response:
44
            generated_tokens.extend(response["token_ids"])
45
46
    print(f"Generated {len(generated_tokens)} tokens: {generated_tokens}")
47
48
if __name__ == "__main__":
49
    asyncio.run(main())

K8s Examples

For basic Kubernetes deployment with the KV Router, see the Kubernetes Deployment section in the Quick Start guide.

Complete K8s Examples

• TRT-LLM aggregated router example
• vLLM aggregated router example
• SGLang aggregated router example
• Distributed inference tutorial

For A/B Testing and Advanced K8s Setup: See the comprehensive KV Router A/B Benchmarking Guide for step-by-step instructions on deploying, configuring, and benchmarking the KV router in Kubernetes.

Example with Advanced Configuration

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
spec:
6
  services:
7
    Frontend:
8
      dynamoNamespace: my-namespace
9
      componentType: frontend
10
      replicas: 1
11
      envs:
12
        - name: DYN_ROUTER_MODE
13
          value: kv
14
        - name: DYN_ROUTER_TEMPERATURE
15
          value: "0.5"  # Add some randomness to prevent worker saturation
16
        - name: DYN_ROUTER_KV_OVERLAP_SCORE_WEIGHT
17
          value: "1.5"  # Prioritize TTFT over ITL
18
        - name: DYN_KV_CACHE_BLOCK_SIZE
19
          value: "16"
20
      extraPodSpec:
21
        mainContainer:
22
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.0

Alternative: Using Command Args in K8s

You can also pass CLI arguments directly in the container command:

1
extraPodSpec:
2
  mainContainer:
3
    image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.0
4
    command:
5
      - /bin/sh
6
      - -c
7
    args:
8
      - "python3 -m dynamo.frontend --router-mode kv --router-temperature 0.5 --http-port 8000"

Recommendation: Use environment variables for easier configuration management and consistency with Dynamo’s K8s patterns.

Routing Patterns

The KvRouter supports multiple usage patterns depending on your control requirements:

1. Automatic Routing (Recommended)

Call generate() directly and let the router handle everything:

1
stream = await router.generate(token_ids=tokens, model="model-name")

• Best for: Most use cases
• Router automatically: Selects best worker, updates state, routes request, tracks lifecycle

2. Manual State Management (Advanced)

Use best_worker(request_id=...) to select and track, then manage the request yourself:

1
worker_id, _dp_rank, overlap = await router.best_worker(tokens, request_id="req-123")
2
response = await client.generate(tokens, request_id="req-123")
3
# await anext(response)  # Get first token
4
await router.mark_prefill_complete("req-123")  # After first token
5
# async for _ in response:  # Continue generating
6
#     ...
7
await router.free("req-123")  # After completion

• Best for: Custom request handling with router state tracking
• Requires: Calling mark_prefill_complete() and free() at correct lifecycle points
• Caution: Incorrect lifecycle management degrades load balancing accuracy

3. Hierarchical Router Probing

Query without state updates, then route through a chosen router:

1
# Probe multiple routers without updating state
2
worker_id_1, dp_rank, overlap_1 = await router_1.best_worker(tokens)  # No request_id
3
worker_id_2, dp_rank, overlap_2 = await router_2.best_worker(tokens)
4
5
# Pick the best router based on results
6
chosen_router = router_1 if overlap_1 > overlap_2 else router_2
7
stream = await chosen_router.generate(tokens, model="model-name", worker_id=worker_id)

• Best for: Multi-tier deployments (e.g., Envoy Gateway routing to multiple router groups)
• Advantage: Query multiple routers before committing to one

4. Custom Load-Based Routing

Use get_potential_loads() to implement custom routing logic:

1
loads = await router.get_potential_loads(tokens)
2
# Apply custom logic (e.g., weighted scoring, constraints)
3
best_worker = min(loads, key=lambda x: custom_cost_fn(x))
4
stream = await router.generate(tokens, model="model-name", worker_id=best_worker['worker_id'])

• Best for: Custom optimization strategies beyond the built-in cost function
• Advantage: Full control over worker selection logic
• See also: Detailed example below in “Custom Routing Example: Minimizing TTFT”

All patterns support router_config_override to adjust routing behavior per-request without recreating the router.

Custom Routing Example: Minimizing TTFT

Here’s an example of using get_potential_loads() to implement custom routing that minimizes Time To First Token (TTFT) by selecting the worker with the least prefill work:

1
import asyncio
2
from dynamo.llm import DistributedRuntime, KvRouter, KvRouterConfig
3
4
async def minimize_ttft_routing():
5
    # Setup router
6
    runtime = DistributedRuntime.detached()
7
    namespace = runtime.namespace("dynamo")
8
    component = namespace.component("backend")
9
    endpoint = component.endpoint("generate")
10
11
    router = KvRouter(
12
        endpoint=endpoint,
13
        block_size=16,
14
        kv_router_config=KvRouterConfig()
15
    )
16
17
    # Your input tokens
18
    token_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
19
20
    # Get potential loads for all workers
21
    potential_loads = await router.get_potential_loads(token_ids)
22
23
    # Find worker with minimum prefill tokens (best for TTFT)
24
    best_worker = min(potential_loads, key=lambda x: x['potential_prefill_tokens'])
25
26
    print(f"Worker loads: {potential_loads}")
27
    print(f"Selected worker {best_worker['worker_id']} with {best_worker['potential_prefill_tokens']} prefill tokens")
28
29
    # Route directly to the selected worker
30
    stream = await router.generate(
31
        token_ids=token_ids,
32
        model="meta-llama/Llama-2-7b-hf",
33
        worker_id=best_worker['worker_id'],  # Force routing to optimal worker
34
        stop_conditions={"max_tokens": 20}
35
    )
36
37
    # Process response
38
    async for response in stream:
39
        if isinstance(response, dict) and "token_ids" in response:
40
            print(f"Generated tokens: {response['token_ids']}")
41
42
if __name__ == "__main__":
43
    asyncio.run(minimize_ttft_routing())

This approach gives you complete control over routing decisions, allowing you to optimize for different metrics based on your specific requirements. As some examples:

• Minimize TTFT: Select worker with lowest potential_prefill_tokens
• Maximize cache reuse: Use best_worker() which considers both prefill and decode loads
• Balance load: Consider both potential_prefill_tokens and potential_decode_blocks together

See Router Design for architecture details and the cost function algorithm.

KV Event Publishing for Custom Engines

For full documentation on implementing KV event publishing for custom inference engines, see the dedicated KV Event Publishing for Custom Engines guide. It covers:

• Direct publishing: Call publish_stored() / publish_removed() to push events over the Dynamo event plane
• ZMQ relay: For engines that emit raw KV events over ZMQ (like vLLM and SGLang), the same KvEventPublisher subscribes to the ZMQ socket and relays events automatically
• API reference, event structure, ZMQ wire format, and best practices

Global Router (Hierarchical Routing)

For deployments with multiple worker pools, the Global Router enables hierarchical routing by sitting between the frontend and local routers. It selects the appropriate pool for each request based on configurable policies, supporting disaggregated topologies where pools are tuned for different workload characteristics.

• Component details: components/src/dynamo/global_router/
• Example: examples/hierarchical_planner/

See Also

• Router README: Quick start guide for the KV Router
• Router Guide: Configuration, tuning, and production setup
• Router Design: Architecture details and event transport modes