DP Rank Routing (Attention Data Parallelism) | NVIDIA Dynamo Documentation

For general TensorRT-LLM features and configuration, see the Reference Guide.

TensorRT-LLM supports attention data parallelism (attention DP) for models like DeepSeek. When enabled, multiple attention DP ranks run within a single worker, each with its own KV cache. Dynamo can route requests to specific DP ranks based on KV cache state.

Dynamo vs TRT-LLM Internal Routing

Dynamo DP Rank Routing: The router selects the optimal DP rank based on KV cache overlap and instructs TRT-LLM to use that rank with strict routing (attention_dp_relax=False). Use this with --router-mode kv for cache-aware routing.
TRT-LLM Internal Routing: TRT-LLM’s scheduler assigns DP ranks internally. Use this with --router-mode round-robin or random when KV-aware routing isn’t needed.

Conversation-Aware DP Rank Routing

To keep a conversation on the same worker and attention-DP rank, enable conversation affinity in both the Dynamo router and the TensorRT-LLM ADP router. Dynamo passes the stable conversation ID to TensorRT-LLM through ConversationParams. Use --conversation-affinity-dp-rank-source to choose which component owns the initial attention-DP rank placement:

Value	Initial placement	Later requests
`engine` (default)	TensorRT-LLM selects the rank with its conversation-aware ADP router.	TensorRT-LLM reuses the stored conversation-to-rank binding.
`dynamo`	Dynamo forwards the KV router’s selected rank as a strict `SchedulingParams.attention_dp_rank`.	TensorRT-LLM reuses the stored conversation-to-rank binding, even if a later Dynamo routing hint differs.

Enable conversation-aware routing in the TensorRT-LLM engine configuration:

1 enable_attention_dp: true
2 attention_dp_config:
3   kv_cache_routing_conversation_affinity: true

Then pass that configuration and configure the rank source on the Dynamo TensorRT-LLM worker:

$ python3 -m dynamo.trtllm \
>   --model-path <MODEL_PATH> \
>   --extra-engine-args engine.yaml \
>   --conversation-affinity \
>   --conversation-affinity-dp-rank-source engine

You can set the same option with the DYN_ENGINE_CONV_AFFINITY_DP_RANK_SOURCE environment variable:

$ export DYN_ENGINE_CONV_AFFINITY_DP_RANK_SOURCE=dynamo

The dynamo rank source requires a TensorRT-LLM version containing NVIDIA/TensorRT-LLM#16815 or equivalent support. Enable it only after Dynamo’s TensorRT-LLM dependency includes that change. Without it, TensorRT-LLM may accept the explicit first-turn rank without recording it as the conversation binding, allowing later turns to move to another rank.

The rank-source option changes only who selects the first rank. Both modes require a stable conversation ID, and TensorRT-LLM owns the conversation-to-rank binding after initial placement.

Enabling DP Rank Routing

$ # Worker with attention DP
$ # (TP=2 acts as the "world size", in effect creating 2 attention DP ranks)
$ CUDA_VISIBLE_DEVICES=0,1 python3 -m dynamo.trtllm \
>   --model-path <MODEL_PATH> \
>   --tensor-parallel-size 2 \
>   --enable-attention-dp \
>   --publish-events-and-metrics
$ 
$ # Frontend with KV routing
$ python3 -m dynamo.frontend --router-mode kv

The --enable-attention-dp flag sets attention_dp_size = tensor_parallel_size and configures Dynamo to publish KV events per DP rank. The router automatically creates routing targets for each (worker_id, dp_rank) combination.

Attention DP requires TRT-LLM’s PyTorch backend. AutoDeploy does not support attention DP.