TensorRT-LLM | NVIDIA Dynamo Documentation

Use the Latest Release

We recommend using the latest stable release of Dynamo to avoid breaking changes.

Dynamo TensorRT-LLM integrates TensorRT-LLM engines into Dynamo’s distributed runtime, enabling disaggregated serving, KV-aware routing, multi-node deployments, and request cancellation. It supports LLM inference, multimodal models, video diffusion, and advanced features like speculative decoding and attention data parallelism.

Experimental Sidecar

The experimental sidecar path runs the Dynamo worker outside TensorRT-LLM and connects through TensorRT-LLM’s native gRPC API. It keeps TensorRT-LLM’s native server and argument surface while separating Dynamo and engine dependencies. See TensorRT-LLM Sidecar for current readiness and a launch example.

Feature Support Matrix

Core Dynamo Features

Feature	TensorRT-LLM	Notes
Disaggregated Serving	✅
Conditional Disaggregation	🚧	Not supported yet
KV-Aware Routing	✅
SLA-Based Planner	✅
Load Based Planner	🚧	Planned
KVBM	✅

Large Scale P/D and WideEP Features

Feature	TensorRT-LLM	Notes
WideEP	✅
DP Rank Routing	✅
GB200 Support	✅

Feature Interactions

TensorRT-LLM delivers maximum inference performance and optimization, with full KVBM integration and robust disaggregated serving support. The matrix below shows which feature pairs are validated to work together.

Legend: ✅ Supported | 🚧 Work in Progress / Experimental / Limited

Feature	Disaggregated Serving	KV-Aware Routing	SLA-Based Planner	KV Block Manager	Multimodal	Request Migration	Request Cancellation	LoRA	Tool Calling	Speculative Decoding
Disaggregated Serving	—
KV-Aware Routing	✅	—
SLA-Based Planner	✅	✅	—
KV Block Manager	✅	✅	✅	—
Multimodal	✅¹	✅²	—	✅	—
Request Migration	✅	✅	✅	✅	🚧	—
Request Cancellation	✅³	✅³	✅³	✅³	✅³	✅³	—
LoRA								—
Tool Calling	✅	✅	✅	✅	✅	✅	✅		—
Speculative Decoding	✅	✅	—	✅	—	✅	✅		✅	—

Notes:

Multimodal Disaggregation: Supports EP/D (Traditional) and E/P/D (Full Disaggregation) image flows, including image URLs and pre-computed embeddings. (Source)

Multimodal + KV-Aware Routing: Image-aware KV routing is supported through the dedicated TRT-LLM MM Router Worker. It requires KV event publishing on the TRT-LLM workers. (Source)

Request Cancellation: Due to known issues, the TensorRT-LLM engine is temporarily not notified of request cancellations, meaning allocated resources for cancelled requests are not freed.

Prerequisites

yq for in-place YAML edits. Install with wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq && chmod +x /usr/local/bin/yq or pip install yq (the latter is a different tool with the same name but similar syntax). If neither is available, a sed fallback is shown inline where yq is used.

Container / driver matrix

Container tag	Backend version	CUDA	Min NVIDIA driver
`tensorrtllm-runtime:1.0.2`	TRT-LLM `v1.3.0rc5.post1`	`v13.1`	`580+`
`vllm-runtime:1.0.2`	vLLM `v0.16.0`	`v12.9`	`575+`
`vllm-runtime:1.0.2-cuda13`	vLLM `v0.16.0`	`v13.0`	`580+`
`sglang-runtime:1.0.2`	SGLang `v0.5.9`	`v12.9`	`575+`
`sglang-runtime:1.0.2-cuda13`	SGLang `v0.5.9`	`v13.0`	`580+`

Source of truth: docs/fern/pages/reference/general/compatibility.mdx and docs/fern/pages/reference/general/release-artifacts.mdx. If those differ from the values above, the source-of-truth files win.

Quick Start

Step 1 (host terminal): Start infrastructure services:

$ docker compose -f dev/docker-compose.yml up -d

Step 2 (host terminal): Pull and run the prebuilt container:

$ DYNAMO_VERSION=1.0.2
$ docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:$DYNAMO_VERSION
$ docker run --gpus all -it --network host --ipc host \
>   nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:$DYNAMO_VERSION

The DYNAMO_VERSION variable above can be set to any specific available version of the container. To find the available tensorrtllm-runtime versions for Dynamo, visit the NVIDIA NGC Catalog for Dynamo TensorRT-LLM Runtime.

Step 3 (inside the container): Launch an aggregated serving deployment (uses Qwen/Qwen3-0.6B by default):

$ cd $DYNAMO_HOME/examples/backends/trtllm
$ ./launch/agg.sh

The launch script will automatically download the model and start the TensorRT-LLM engine. You can override the model by setting MODEL_PATH and SERVED_MODEL_NAME environment variables before running the script.

Step 4 (host terminal): Verify the deployment:

$ curl localhost:8000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "Qwen/Qwen3-0.6B",
>     "messages": [{"role": "user", "content": "Explain why Roger Federer is considered one of the greatest tennis players of all time"}],
>     "stream": true,
>     "max_tokens": 30
>   }'

Deploy

Deploy TensorRT-LLM with Dynamo on Kubernetes using a DynamoGraphDeployment. Before kubectl apply, substitute the container image tag in the deployment YAML. The sed fallback is shown inline for environments without yq:

$ # yq
$ yq -i '(.spec.services[].extraPodSpec.mainContainer.image) |= sub(":1\.0\.2", ":<your-tag>")' deploy.yaml
$ # sed fallback
$ sed -i.bak 's|:1\.0\.2|:<your-tag>|g' deploy.yaml

For full Kubernetes deployment instructions, see the TensorRT-LLM Kubernetes Deployment Guide.

Next Steps

Reference Guide: Features, configuration, and operational details
Examples: Local deployment launch scripts
KV Cache Transfer: KV cache transfer methods for disaggregated serving
Observability: Metrics and monitoring
Multinode Examples: Multi-node deployment with SLURM
Deploying TensorRT-LLM with Dynamo on Kubernetes: Kubernetes deployment guide