DP Rank Routing (Attention Data Parallelism)

For general TensorRT-LLM features and configuration, see the Reference Guide.

TensorRT-LLM supports attention data parallelism (attention DP) for models like DeepSeek. When enabled, multiple attention DP ranks run within a single worker, each with its own KV cache. Dynamo can route requests to specific DP ranks based on KV cache state.

Dynamo vs TRT-LLM Internal Routing

• Dynamo DP Rank Routing: The router selects the optimal DP rank based on KV cache overlap and instructs TRT-LLM to use that rank with strict routing (attention_dp_relax=False). Use this with --router-mode kv for cache-aware routing.
• TRT-LLM Internal Routing: TRT-LLM’s scheduler assigns DP ranks internally. Use this with --router-mode round-robin or random when KV-aware routing isn’t needed.

Conversation-Aware DP Rank Routing

To keep a conversation on the same worker and attention-DP rank, enable conversation affinity in both the Dynamo router and the TensorRT-LLM ADP router. Dynamo passes the stable conversation ID to TensorRT-LLM through ConversationParams.

Enable conversation-aware routing in the TensorRT-LLM engine configuration:

1
enable_attention_dp: true
2
attention_dp_config:
3
  kv_cache_routing_conversation_affinity: true

Then pass that configuration and enable conversation affinity on the Dynamo TensorRT-LLM worker:

$
python3 -m dynamo.trtllm \
>
  --model-path <MODEL_PATH> \
>
  --extra-engine-args engine.yaml \
>
  --conversation-affinity

Conversation affinity requires a stable conversation ID, and TensorRT-LLM owns the conversation-to-rank binding after initial placement.

Enabling DP Rank Routing

$
# Worker with attention DP
$
# (TP=2 acts as the "world size", in effect creating 2 attention DP ranks)
$
CUDA_VISIBLE_DEVICES=0,1 python3 -m dynamo.trtllm \
>
  --model-path <MODEL_PATH> \
>
  --tensor-parallel-size 2 \
>
  --enable-attention-dp \
>
  --publish-events-and-metrics
$
$
# Frontend with KV routing
$
python3 -m dynamo.frontend --router-mode kv

The --enable-attention-dp flag sets attention_dp_size = tensor_parallel_size and configures Dynamo to publish KV events per DP rank. The router automatically creates routing targets for each (worker_id, dp_rank) combination.