TensorRT-LLM Sidecar | NVIDIA Dynamo Documentation

Experimental. The sidecars and their deployment examples are experimental. Manifests, flags, and behavior may change without notice.

dynamo-trtllm-sidecar connects a Dynamo worker to TensorRT-LLM’s OpenEngine gRPC server. See Sidecar Backends for the architecture.

For the best and latest support, use the upstream TensorRT-LLM nightly image, which carries the latest gRPC server updates. Nightly builds are the nvcr.io/nvidia/tensorrt-llm/release:<version>.dev<date> tags on NGC.

Support Matrix

Feature	Supported
Aggregated	Yes
Disaggregated	Yes
KV routing	No

Launch Locally

See lib/sidecar/trtllm/launch/ for all topologies. For example, aggregated serving on one GPU:

export DYN_DISCOVERY_BACKEND=file   # single host: no etcd or NATS needed
./lib/sidecar/trtllm/launch/agg.sh

In a second terminal:

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Deploy on Kubernetes

See lib/sidecar/trtllm/deploy/ for all manifests. For example, aggregated serving:

Before applying, replace <your-registry>/dynamo-sidecar in the manifest with a sidecar image and create the hf-token-secret Secret that the manifest reads.

kubectl create secret generic hf-token-secret -n <namespace> \
  --from-literal=HF_TOKEN=<your-hf-token>
kubectl apply -f lib/sidecar/trtllm/deploy/agg.yaml -n <namespace>
kubectl port-forward -n <namespace> svc/trtllm-sidecar-agg-frontend 8000:8000

Topologies

The frontend reaches each sidecar over Dynamo’s request, discovery, and event planes; the sidecar reaches the engine over its native gRPC API. The TensorRT-LLM sidecar does not publish KV cache events.

Single-Node TP

One engine on one node, with one sidecar.

On one node, a request reaches the TensorRT-LLM tensor-parallel ranks through the Dynamo Sidecar. The Dynamo Frontend sends requests over the request plane to the sidecar.

Multi-Node TP

One engine spans two nodes. Only the leader node has a sidecar; the follower node holds the remaining TP ranks.

When one TensorRT-LLM engine spans two nodes with tensor parallelism, only the leader node runs a Dynamo Sidecar. The Dynamo Frontend sends requests over the request plane to the sidecar on Node 0.

Multi-Node DP

Not supported yet. The TensorRT-LLM sidecar does not target DP ranks or publish KV events.

TensorRT-LLM data parallelism across two nodes. Only the leader sidecar serves requests: the Dynamo Frontend router picks a DP rank and sends requests over the request plane to the node 0 Dynamo Sidecar, which registers DP ranks 0-3 and calls its local TensorRT-LLM over native gRPC.