> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # TLS NVIDIA Dynamo supports opt-in TLS encryption for the frontend HTTP API, the TCP request and response streams between frontends and workers, and connections to NATS. On the TCP transport, both the **request plane** (frontend → worker inference requests) and the **response stream** (worker → frontend inference output) are encrypted using [rustls](https://github.com/rustls/rustls) with the `ring` cryptographic provider. When no TLS configuration is provided, these transports operate in plaintext exactly as before. The `DYN_TCP_TLS_*` environment variables encrypt the frontend↔worker TCP request and response streams; NATS traffic is encrypted separately via NATS TLS (see the NATS TLS section below). The KV event plane is a separate transport: when it runs over ZMQ it is **not** encrypted, and when it runs over NATS it is encrypted only if NATS TLS is configured. ## HTTP TLS and mTLS The frontend accepts HTTPS connections from external clients or an API gateway. Configure these options on the frontend; they are independent of the internal `DYN_TCP_TLS_*` and `NATS_TLS_*` settings. | Environment variable | Frontend CLI flag | Description | | ----------------------------- | --------------------------- | ---------------------------------------------------------------------------------------- | | `DYN_TLS_CERT_PATH` | `--tls-cert-path` | PEM server certificate. Set together with the server private key to enable HTTPS. | | `DYN_TLS_KEY_PATH` | `--tls-key-path` | PEM private key for the server certificate. | | `DYN_TLS_CLIENT_CA_CERT_PATH` | `--tls-client-ca-cert-path` | PEM CA certificates trusted to sign client certificates. Enables HTTP mutual TLS (mTLS). | HTTP mTLS requires both the server certificate and private key. When a client CA is configured, the frontend rejects TLS handshakes from clients that omit a certificate or present one that is not trusted by that CA. Without a client CA, HTTPS authenticates only the server. The client or API gateway configures its own client certificate/key and trust for the frontend's server certificate. The frontend loads its HTTP TLS configuration at startup. Its server certificate and private key hot-reload for subsequent handshakes; changing the trusted client CA requires a restart. See [Design notes](#design-notes) for rotation behavior and the [frontend configuration reference](/dynamo/dev/reference/components/frontend-configuration) for the complete frontend argument list. ## Environment variables The following variables configure internal TCP TLS. Server configuration is validated when the server starts; client connectors initialize lazily on the first outbound connection. Both frontends and workers act as TCP server and client depending on the stream direction (response streams: worker dials frontend; request streams: frontend dials worker). Set the internal TCP TLS configuration on every participating pod. HTTP TLS settings apply only to the frontend HTTP server. ### Server role (accepting connections) | Variable | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------ | | `DYN_TCP_TLS_CERT_PATH` | Path to the PEM certificate file. When set together with `DYN_TCP_TLS_KEY_PATH`, TLS is enabled on the TCP server. | | `DYN_TCP_TLS_KEY_PATH` | Path to the PEM private key for the server certificate. | ### Client role (dialing connections) | Variable | Description | | ------------------------------------ | -------------------------------------------------------------------------------------------------------- | | `DYN_TCP_TLS_CA_CERT_PATH` | Path to the PEM CA certificate used to verify the peer's server certificate. | | `DYN_TCP_TLS_INSECURE` | Set to `1` or `true` to skip certificate verification. For local development only. | | `DYN_TCP_TLS_SERVER_NAME` | Override the TLS SNI hostname. Useful when connecting by IP to a server whose certificate has a DNS SAN. | | `DYN_TCP_TLS_HANDSHAKE_TIMEOUT_SECS` | TLS handshake timeout in seconds (default: 3). | ## CLI flags The same configuration is available via command-line flags on all backends (vllm, sglang, trtllm, tokenspeed) through `DynamoRuntimeArgGroup`: ``` --tcp-tls-cert-path PATH Server certificate (PEM) --tcp-tls-key-path PATH Server private key (PEM) --tcp-tls-ca-cert-path PATH CA certificate for server verification (PEM) --tcp-tls-insecure Disable certificate verification --tcp-tls-server-name NAME Override TLS SNI hostname --tcp-tls-handshake-timeout N Handshake timeout in seconds (default: 3) ``` The frontend (`dynamo.frontend`) also accepts `--tcp-tls-cert-path`, `--tcp-tls-key-path`, and `--tcp-tls-ca-cert-path`. ## Quick start Generate a self-signed certificate for local testing: ```bash # Generate CA openssl req -x509 -newkey rsa:2048 -keyout ca-key.pem -out ca-cert.pem \ -days 365 -nodes -subj "/CN=DynamoCA" # Generate server cert with SAN openssl req -newkey rsa:2048 -keyout server-key.pem -out server-csr.pem \ -nodes -subj "/CN=localhost" \ -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" openssl x509 -req -in server-csr.pem -CA ca-cert.pem -CAkey ca-key.pem \ -CAcreateserial -out server-cert.pem -days 365 -copy_extensions copyall ``` Both frontend and worker need the same flags (both act as server and client): ```bash python -m dynamo.vllm \ --tcp-tls-cert-path server-cert.pem \ --tcp-tls-key-path server-key.pem \ --tcp-tls-ca-cert-path ca-cert.pem \ --tcp-tls-server-name localhost \ ... python -m dynamo.frontend \ --tcp-tls-cert-path server-cert.pem \ --tcp-tls-key-path server-key.pem \ --tcp-tls-ca-cert-path ca-cert.pem \ --tcp-tls-server-name localhost \ ... ``` ## Kubernetes deployment In Kubernetes, TLS certificates are typically delivered by a certificate management system and mounted into pods. Set the environment variables on each component's pod template in the `DynamoGraphDeployment` spec: ```yaml spec: components: - name: Frontend podTemplate: spec: containers: - name: main env: - name: DYN_TCP_TLS_CERT_PATH value: /etc/certs/server/cert.pem - name: DYN_TCP_TLS_KEY_PATH value: /etc/certs/server/key.pem - name: DYN_TCP_TLS_CA_CERT_PATH value: /etc/certs/ca/ca.pem - name: worker podTemplate: spec: containers: - name: main env: - name: DYN_TCP_TLS_CERT_PATH value: /etc/certs/server/cert.pem - name: DYN_TCP_TLS_KEY_PATH value: /etc/certs/server/key.pem - name: DYN_TCP_TLS_CA_CERT_PATH value: /etc/certs/ca/ca.pem ``` Both components need the same TLS env vars because each acts as both TCP server and client depending on the stream direction. For platform-wide TLS that the operator injects into every deployment automatically — instead of setting env vars on each component — see [Operator TLS](/dynamo/dev/kubernetes/installation/operator-tls). ## NATS TLS NATS carries JetStream indexer recovery/replay, the audit sink, and — when the request plane is set to NATS (`--request-plane nats`) — inference request distribution. TLS is configured separately from TCP: | Variable | Description | | ----------------------- | ----------------------------------------------------------------------------------- | | `NATS_TLS_CA_CERT_PATH` | CA certificate to verify the NATS server. When set, a custom TLS config is applied. | | `NATS_TLS_INSECURE` | Skip NATS server certificate verification (dev only). | When only the `tls://` URL scheme is used without explicit TLS env vars, async-nats handles TLS natively with system roots. The `NATS_SERVER` URL accepts both `nats://` and `tls://` schemes, but setting any explicit NATS TLS variable (`NATS_TLS_*`) requires the `tls://` scheme — startup fails with a clear error otherwise. CLI flags (all backends via `DynamoRuntimeArgGroup`): ```text --nats-tls-ca-cert-path PATH CA certificate for NATS server verification (PEM) --nats-tls-insecure Disable NATS certificate verification ``` The frontend (`dynamo.frontend`) also accepts `--nats-tls-ca-cert-path` and `--nats-tls-insecure`. ### Enabling TLS on the NATS server The env vars above configure the **client** side (Dynamo verifying the NATS server). For TLS to work, the NATS server itself must also be configured to listen on TLS — otherwise a client that sets `NATS_TLS_CA_CERT_PATH` (or the operator-level `natsTLSCAPath`) will attempt a TLS handshake against a plaintext port and fail. When deploying Dynamo's platform chart, the bundled NATS subchart exposes this via `nats.config.nats.tls`. The certificates are typically delivered by a certificate management system such as cert-manager. **One-way TLS** (server presents a cert, clients verify it): ```yaml # NATS server side — platform chart values nats: config: nats: tls: enabled: true secretName: nats-server-tls # Secret with tls.crt / tls.key (e.g. cert-manager) ``` Or, if the cert files are mounted into the pod by an external system, use the `merge` block to point at the on-disk paths directly: ```yaml nats: config: nats: tls: enabled: true merge: cert_file: /etc/certs/server/cert.pem key_file: /etc/certs/server/key.pem timeout: 2 ``` Then point the Dynamo clients at the TLS endpoint and give them the CA. These are operator-subchart values, so nest them under `dynamo-operator:` when using the platform chart: ```yaml dynamo-operator: natsAddr: "tls://dynamo-platform-nats.dynamo-system.svc.cluster.local:4222" natsTLSCAPath: /etc/certs/ca/ca.pem # → NATS_TLS_CA_CERT_PATH ``` **mTLS** (server also verifies client certs) — add `ca_file` and `verify` to the server's `merge` block, and present a client identity from the Dynamo side: ```yaml # NATS server side nats: config: nats: tls: enabled: true merge: cert_file: /etc/certs/server/cert.pem key_file: /etc/certs/server/key.pem ca_file: /etc/certs/client-ca/ca.pem # verify client certs verify: true # require client certs ``` ```yaml # Dynamo client side (operator-level, nested under dynamo-operator:) dynamo-operator: natsTLSCAPath: /etc/certs/ca/ca.pem natsTLSClientCertPath: /etc/certs/client/cert.pem natsTLSClientKeyPath: /etc/certs/client/key.pem ``` See the [NATS TLS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) for the full server-side TLS schema. ## Mutual TLS (mTLS) By default only the server is authenticated (the client verifies the server's certificate). Mutual TLS additionally makes the **client present a certificate** that the **server verifies**, so both ends of a connection prove their identity. mTLS is opt-in and layered on top of the TLS configuration above. For the **TCP** transports (request plane + response stream) Dynamo owns both ends, so configure two things and — since every component is both TCP client and server — set both on every pod: * **On the client:** a client certificate/key to present. * **On the server:** a CA certificate (`DYN_TCP_TLS_CLIENT_CA_CERT_PATH`) to verify the presented client certificate. When set, the server **requires** a trusted client certificate and rejects the handshake otherwise. For **NATS** the broker is external, so Dynamo only presents a client identity — there is no Dynamo-side "server" half. Whether client certificates are actually required is enforced by the **NATS server** configuration (e.g. `tls { ca_file: ...; verify: true }`), not by Dynamo. ### TCP request/response streams | Variable | Role | Description | | --------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------- | | `DYN_TCP_TLS_CLIENT_CERT_PATH` | Client | PEM client certificate presented to the server. Set with `DYN_TCP_TLS_CLIENT_KEY_PATH`. | | `DYN_TCP_TLS_CLIENT_KEY_PATH` | Client | PEM private key for the client certificate. | | `DYN_TCP_TLS_CLIENT_CA_CERT_PATH` | Server | PEM CA used to verify client certificates. When set, clients **must** present a certificate signed by this CA. | ### NATS | Variable | Role | Description | | --------------------------- | ------ | ----------------------------------------------------------------------------------------- | | `NATS_TLS_CLIENT_CERT_PATH` | Client | PEM client certificate presented to the NATS server. Set with `NATS_TLS_CLIENT_KEY_PATH`. | | `NATS_TLS_CLIENT_KEY_PATH` | Client | PEM private key for the NATS client certificate. | > **Note:** Dynamo cannot detect whether the NATS broker actually requests a > client certificate. If you set `NATS_TLS_CLIENT_CERT_PATH`/`_KEY_PATH` but the > broker is not configured to verify client certs, the connection silently > falls back to ordinary one-way TLS. Enable verification on the NATS server > (`verify: true`) to make NATS mTLS effective. Client certificate and key must be set **together**, and a client identity requires a server CA (`*_CA_CERT_PATH`) — or insecure mode for development — so the server can still be verified. The TCP **servers** and the **NATS** client validate this at startup and fail closed. The TCP **client** connectors, however, initialize lazily on the first outbound connection, so an incomplete TCP client config surfaces as a per-connection handshake failure rather than a startup crash. The client certificate/key hot-reload from disk on rotation, exactly like the server certificate (see Design notes). The same options are available as CLI flags on all backends (via `DynamoRuntimeArgGroup`) and the frontend (`dynamo.frontend`): ```text --tcp-tls-client-cert-path PATH Client certificate presented for TCP mTLS (PEM) --tcp-tls-client-key-path PATH Client private key for TCP mTLS (PEM) --tcp-tls-client-ca-cert-path PATH CA to verify client certs; enforces TCP mTLS (PEM) --nats-tls-client-cert-path PATH Client certificate presented for NATS mTLS (PEM) --nats-tls-client-key-path PATH Client private key for NATS mTLS (PEM) ``` ## Encrypted paths When TLS is configured, the following transports are encrypted (the ZMQ event plane is not covered): | Path | Direction | Data | Transport | | --------------- | ----------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------- | | HTTP API | Client/API gateway → Frontend | API requests and responses | HTTPS; optional client-certificate authentication | | Request plane | Frontend → Worker | User prompts, request metadata | `egress/tcp_client` → `ingress/shared_tcp_endpoint` | | Response stream | Worker → Frontend | Inference output tokens | `tcp/client` → `tcp/server` | | Request stream | Frontend → Worker | Streaming input (bidirectional) | `tcp/client` → `tcp/server` | | NATS | Frontend/Worker ↔ NATS | Inference requests (when `--request-plane nats`), JetStream recovery, audit logs | `transports/nats` | ## Design notes * HTTP server certificates and private keys use the shared hot-reload resolver. A background thread checks for file changes every 30 seconds (sooner after a failed reload). Valid updates apply to subsequent handshakes; existing connections remain open, and a failed reload retains the last valid certificate. The HTTP client CA is loaded at startup and requires a frontend restart after rotation. * Server certificates hot-reload: the request-plane and response-stream servers serve their leaf cert/key through a resolver that re-reads the files from disk when their contents change (detected by a content hash, so rotations done by an atomic symlink swap are handled too), so certificate rotation takes effect **without a process restart**. The check is rate-limited (at most once every 30s, sooner after a failed reload) and never blocks a handshake; a failed reload keeps serving the last valid certificate. Client trust anchors (the CA) are still loaded once, so rotating the CA itself requires a restart. * mTLS **client identity** certificates hot-reload through the same resolver, so a rotated client cert/key is picked up without a restart. The CA used by the server to verify client certificates is loaded once (rotating it needs a restart). * Client TLS connectors are built once and cached via `OnceCell` on the first outbound connection. * The TLS handshake is spawned per-connection on both the request plane and response stream servers so the accept loop is never blocked. * Invalid TLS configuration on the request plane (e.g. bad cert path) prevents server startup rather than silently falling back to plaintext. * When server and client TLS configurations are mismatched (e.g., server has TLS but client does not), a warning is logged at startup. * An empty CA certificate file is detected at load time and rejected with a clear error message.