TensorRT-LLM Sidecar

dynamo-trtllm-sidecar connects a Dynamo worker to TensorRT-LLM’s OpenEngine gRPC server. See Sidecar Backends for the architecture.

Support Matrix

Launch Locally

See lib/sidecar/trtllm/launch/ for all topologies. For example, aggregated serving on one GPU:

export DYN_DISCOVERY_BACKEND=file   # single host: no etcd or NATS needed
./lib/sidecar/trtllm/launch/agg.sh

In a second terminal:

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Deploy on Kubernetes

See lib/sidecar/trtllm/deploy/ for all manifests. For example, aggregated serving:

Before applying, replace <your-registry>/dynamo-sidecar in the manifest with a sidecar image and create the hf-token-secret Secret that the manifest reads.

kubectl create secret generic hf-token-secret -n <namespace> \
  --from-literal=HF_TOKEN=<your-hf-token>
kubectl apply -f lib/sidecar/trtllm/deploy/agg.yaml -n <namespace>
kubectl port-forward -n <namespace> svc/trtllm-sidecar-agg-frontend 8000:8000

Multinode deployments require complete leader and worker role PodTemplates. Set backendFramework: trtllm as the examples do: the operator does not infer a backend from a native engine launch, and on Grove it starts the leader clique after the worker clique only when the backend is known. The templates own the MPI launch, the SSH setup, the engine probes, and the sidecar placement. In sidecar mode the operator does not apply its TensorRT-LLM SSH key mount, mpirun wrapper, or worker probe rewrite. See the TP example, where the leader launches every rank with mpirun over SSH and runs the only sidecar, and the attention-DP example, where every rank publishes KV events on its own node, so the worker also runs a sidecar that relays its local events under the leader’s worker identity through OpenEngine’s follower metadata endpoint. Engine-failure KV recovery remains incomplete and is tracked in DEP #14897.

The leader runs trtllm-llmapi-launch trtllm-serve --grpc --grpc-protocol openengine --host 0.0.0.0 and uses Kubernetes gRPC probes against the standard health service: liveness for the startup and liveness probes, inference for readiness. That service is in TensorRT-LLM nightlies from 2026-10-27; node-local discovery and native KV events need 1.4.0rc1 or later. Worker main containers run sshd for the MPI launch and carry no probes: the leader waits for sshd, and losing sshd or a rank aborts mpirun and restarts the leader. The release images ship no sshd, so the examples install openssh-server at start, and they export the container environment to sshd sessions so Triton JIT on the worker finds CUDA. TensorRT-LLM attention DP requires untied input and output embeddings; the DP example serves TinyLlama/TinyLlama-1.1B-Chat-v1.0.

The examples use DYNAMO_RANK and DYNAMO_LEADER_ADDRESS, which exist only in DGDs created by operator 1.6.0 or later. For an older DGD, see Use the variables in a DGD created before 1.6.0.

Topologies

The frontend sends inference requests to each engine’s leader sidecar, which forwards them to OpenEngine over gRPC and polls routing load for all attention-DP ranks. Each sidecar discovers its node’s KV event sources through OpenEngine and relays local KV events through Dynamo’s event plane. Follower sidecars do not serve inference requests.

Single-Node TP

One engine on one node, with one sidecar.

Multi-Node TP

One engine spans two nodes. Only the leader node has a sidecar; the follower node holds the remaining TP ranks.

Multi-Node DP

Every rank publishes KV events on its own node. The leader sidecar registers all ranks and serves requests; a worker sidecar relays its node-local events without serving. The sidecar forwards the selected attention-DP rank to OpenEngine for KV-aware routing when the engine has more than one attention-DP rank. For single-rank engines, it omits the redundant rank hint and does not synthesize a conversation ID for anonymous requests. Explicit session IDs are preserved.