> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Runtime Configuration (DynamoRuntimeConfig) `DynamoRuntimeConfig` holds the configuration that is **common to every Dynamo process** — the frontend, the standalone router, and all three backend workers (vLLM, SGLang, TRT-LLM) parse this same argument group. Most fields, types, defaults, and choices on this page come from the [`DynamoRuntimeArgGroup` and `DynamoRuntimeConfig`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/common/configuration/groups/runtime_args.py) definitions. The page also catalogs runtime environment-only controls implemented below the Python argument layer. > **Note** > > These are the **shared** Dynamo runtime flags. Each backend also has its own backend-specific flags: [vLLM Configuration](/dynamo/dev/reference/backends/v-llm-configuration), [SGLang Configuration](/dynamo/dev/reference/backends/sg-lang-configuration), and [TensorRT-LLM Configuration](/dynamo/dev/reference/backends/tensor-rt-llm-configuration). This page covers neither native engine arguments nor backend-specific `DYN_*` prefixed flags — only the flags from `DynamoRuntimeArgGroup`. ## How the config is loaded Unless a field is marked environment-only, it has both a CLI flag and an environment variable. The CLI flag takes precedence; the environment variable is the fallback. Boolean fields are negatable — for example, `--enable-local-indexer` sets it on and `--no-enable-local-indexer` sets it off. #### Kubernetes Set flags in **any** component container's `args` and environment variables in its `env`, under a service of a [DynamoGraphDeployment](/dynamo/dev/reference/api/kubernetes/dynamo-graph-deployment) (DGD). To set a `DYN_*` variable across **all** components at once, use the graph-level `spec.envs` field. ```yaml spec: # Graph-level: applies to every component in the deployment envs: - name: DYN_ENDPOINT_TYPES value: chat services: worker: extraPodSpec: mainContainer: args: - --dyn-tool-call-parser - llama3_json - --engine-request-limit - "32" env: - name: DYN_MULTIMODAL_EMBEDDING_CACHE_CAPACITY_GB value: "4.0" ``` On Kubernetes, several of these variables are **auto-injected** by the Dynamo operator into every container — see the individual field notes below. #### Local Pass the flags directly on the command line of any Dynamo component: ```bash # Frontend python -m dynamo.frontend \ --namespace my-cluster \ --discovery-backend etcd \ --endpoint-types chat # vLLM worker python -m dynamo.vllm \ --namespace my-cluster \ --model meta-llama/Llama-3.1-8B-Instruct \ --dyn-tool-call-parser llama3_json \ --exclude-tools-when-tool-choice-none # SGLang worker python -m dynamo.sglang \ --namespace my-cluster \ --model-path Qwen/Qwen3-32B ``` ## Model discovery and namespace **`--namespace`** `string` — default: dynamo Dynamo namespace that scopes service discovery. All components in a deployment must share the same namespace to find each other. If `DYN_NAMESPACE_WORKER_SUFFIX` is also set, `-{suffix}` is automatically appended to the resolved value to support multiple worker pools serving the same model. On Kubernetes the operator sets this automatically to `{k8s_namespace}-{dgd_name}`; override only deliberately. Environment variable: `DYN_NAMESPACE` --- **`--endpoint`** `string` — default: null Dynamo endpoint string in `dyn://namespace.component.endpoint` format, for example `dyn://dynamo.backend.generate`. When unset, the endpoint address is inferred from the component's registration in the discovery backend. Environment variable: `DYN_ENDPOINT` --- **`--endpoint-types`** `string` — default: chat,completions Comma-separated list of OpenAI-compatible endpoint types to enable. Use `completions` alone for models that do not have a chat template. The obsolete alias `--dyn-endpoint-types` is accepted for backward compatibility. Environment variable: `DYN_ENDPOINT_TYPES` --- ## Communication planes **`--discovery-backend`** `string` — default: etcd Service discovery backend. `kubernetes` uses the K8s API; `etcd` uses a distributed key-value store (configured via `ETCD_*` env vars such as `ETCD_ENDPOINTS`); `file` uses the local filesystem (path from `DYN_FILE_KV`, defaulting to `$TMPDIR/dynamo_store_kv`); `mem` uses an in-memory store suitable for single-process development. See [Discovery Plane](/dynamo/dev/knowledge-base/concepts/architecture#discovery-plane) for design details. On Kubernetes the operator sets this automatically to `kubernetes`; override only deliberately. Allowed values: kubernetes etcd file mem Environment variable: `DYN_DISCOVERY_BACKEND` --- **`NATS_STARTUP_CONNECT_TIMEOUT_SECONDS`** `integer` — default: 120 Maximum number of seconds that a process spends establishing its initial NATS connection. Applies only to processes that connect to NATS. Dynamo retries DNS, I/O, and timeout failures with exponential backoff from 1 second up to 5 seconds until this deadline; each attempt times out after at most 5 seconds. Authentication, authorization, TLS, and invalid server address errors fail immediately. Set a larger value when NATS can take longer to become available during deployment startup or recovery; set a smaller value when the process should fail fast. On Kubernetes, ensure the pod's startup probe allows at least this long. Values must be greater than `0`. Zero or invalid values log a warning and fall back to `120`. This setting does not affect reconnects after startup, which use a fixed 5-second per-attempt timeout. --- **`ETCD_STARTUP_CONNECT_TIMEOUT_SECONDS`** `integer` — default: 120 Maximum number of seconds that an etcd discovery client spends establishing its initial connection and primary lease. Dynamo retries with exponential backoff until this deadline. Set a larger value when etcd can take longer to become available during deployment startup or recovery; set a smaller value when the process should fail fast. Values must be greater than `0`. Zero or invalid values log a warning and fall back to `120`. --- **`--request-plane`** `string` — default: tcp Transport used to distribute requests from routers to workers. `tcp` provides the lowest latency and is recommended for production. `nats` uses NATS messaging. See [Request Plane](/dynamo/dev/knowledge-base/concepts/architecture#request-plane) for design details. Allowed values: tcp nats Environment variable: `DYN_REQUEST_PLANE` --- **`DYN_REQUEST_PLANE_CODEC`** `string` — default: msgpack Preferred payload codec advertised by every request-plane endpoint served by this process. Clients select the codec independently for each destination endpoint. They use the destination's advertised codec, or `json` when a legacy destination does not advertise one. Setting this variable does not force the codec for outbound requests from the process. Dynamo caches this process-wide value on its first codec lookup. Restart the process after changing it. Allowed values: json msgpack --- **`DYN_TCP_MAX_MESSAGE_SIZE`** `integer` — default: 33554432 Maximum TCP request-plane frame size in bytes (default 32 MiB). The frontend rejects an oversized frame with HTTP 400 before writing it. This limit is independent of `DYN_HTTP_BODY_LIMIT_MB` (default 192 MB) on the HTTP ingress: a chat-completions body can be accepted by the HTTP server and still fail at this cap after serialization. Inline `data:` media is sent once in `multi_modal_data`; raise this value when operators must accept larger inline-media worker frames. See [Frontend Configuration](/dynamo/dev/reference/components/frontend-configuration) for `DYN_HTTP_BODY_LIMIT_MB`. --- **`DYN_TCP_RPC_HOST`** `string` — default: auto-detected local address Host for the TCP request-plane server. Accepts IPv4 and IPv6 literals, bracketed IPv6 literals, wildcards, and interface names, including aliases such as `eth0:1`. If unset or empty, Dynamo selects a usable non-loopback IPv4 address, then IPv6, then IPv4 loopback, then IPv6 loopback. Within each family, it prefers the address the operating system uses for outbound traffic (on Linux, the preferred source address of the route to an external address) when an up interface carries it, and otherwise the first usable address in operating-system enumeration order. On Unix, automatic selection excludes interfaces that are down. Dynamo trims surrounding whitespace from configured hosts; a value containing only whitespace still fails server startup. For a named interface, Dynamo uses the same address selection order, restricted to that interface. It selects the first usable address in each category in operating-system enumeration order. An interface with only loopback addresses uses IPv4 loopback, then IPv6 loopback. Explicit interface lookup does not filter by interface state. A wildcard uses the same automatic interface inventory to select a usable non-loopback address in its requested family for advertisement. If only the other family has one, Dynamo switches the bind wildcard to that family. If neither family has one, Dynamo prefers loopback in the requested family before trying the other family. If no loopback address is found, it uses the requested family's standard loopback address. The bind wildcard uses the selected address family. Wildcards are never advertised. IPv4 link-local addresses are accepted. Multicast, broadcast, and IPv6 link-local addresses are rejected. IPv4-mapped IPv6 addresses use IPv4 rules. For example, `::ffff:0.0.0.0` uses the `0.0.0.0` wildcard rules and advertises a concrete address. The server resolves the address once at startup. A loopback fallback persists until restart, and an interface-enumeration error fails server startup. --- **`DYN_TCP_RPC_PORT`** `integer` — default: 0 Port used by the TCP request-plane server. Accepted values are integers from `0` to `65535`. When the value is unset, empty, or `0`, the operating system assigns a free port. Dynamo does not trim whitespace. An invalid value, including a value with surrounding whitespace, silently falls back to an operating-system-assigned port. --- **`DYN_TCP_RESPONSE_STREAM_HOST`** `string` — default: auto-detected local address Host for the TCP response-stream server. It accepts the same address syntax and uses the same address selection and wildcard-family rules as `DYN_TCP_RPC_HOST`. Dynamo trims whitespace and treats an empty value as unset. With a wildcard host, the TCP listener binds the selected wildcard, while the QUIC response listener binds only the selected concrete address. The server resolves the address once at startup. A loopback fallback persists until restart, and an interface-enumeration error fails server startup. --- **`DYN_TCP_RESPONSE_STREAM_PORT`** `integer` — default: 0 Port used by the TCP response stream server. Dynamo trims surrounding whitespace. When the value is unset, empty, or `0`, the operating system assigns a free port. Values must be between `0` and `65535`. --- **`--event-plane`** `string` — default: zmq Event publishing transport. ZMQ is the default for every discovery backend; select `nats` explicitly to use NATS Core. See [Event Plane](/dynamo/dev/knowledge-base/concepts/architecture#event-plane) for design details. Allowed values: nats zmq Environment variable: `DYN_EVENT_PLANE` --- **`DYN_EVENT_PLANE_HOST`** `string` — default: auto-detected local address IPv4 or IPv6 address, or exact network interface name, advertised by direct ZMQ event publishers. For an interface name, Dynamo selects the first usable IPv4 address, then IPv6. Dynamo trims whitespace and treats an empty value as unset. Unspecified addresses such as `0.0.0.0` and `::` are invalid, including IPv4-mapped wildcards such as `::ffff:0.0.0.0`. When unset, Dynamo selects a usable non-loopback IPv4 address first, then IPv6, preferring the operating system's routed source address as described for `DYN_TCP_RPC_HOST`. On Unix, automatic selection excludes down interfaces. If neither family has a usable non-loopback address, Dynamo selects IPv4 loopback, then IPv6 loopback, and falls back to `127.0.0.1` if neither is found. Remote subscribers cannot reach a loopback address. Automatic selection can choose an IPv4 link-local address or a bridge address. On hosts with several interfaces, set this variable to a reachable IP address or interface if automatic selection chooses the wrong address. Direct ZMQ publishers bind `0.0.0.0` for an advertised IPv4 address or `[::]` for an advertised IPv6 address. Discovery publishes the concrete address with the port assigned to that listener. An explicit override must identify an address reachable by subscribers. The IPv6 wildcard can also accept IPv4 connections; keep ZMQ listeners on trusted networks. Event-plane PUB/SUB sockets support IPv4 addresses and bracketed IPv6 addresses, including connections to configured ZMQ brokers. Hostnames retain IPv4 resolution. For an IPv6 broker, use a bracketed address such as `tcp://[2001:db8::10]:5555`; Dynamo rejects an unbracketed IPv6 address. This variable does not affect NATS or ZMQ broker mode. --- > **Note** > > On IPv6-only hosts, set `DYN_HTTP_HOST=::` on the Frontend and `DYN_SYSTEM_HOST=::` on workers. Both > listeners default to `0.0.0.0`, which accepts IPv4 connections only. The request plane and the direct > ZMQ event plane select an IPv6 address automatically when the host has no usable non-loopback IPv4 > address. **`DYN_ZMQ_IO_THREADS`** `integer` — default: 4 Number of libzmq background I/O threads in the process-wide event-plane context. All event-plane PUB/SUB sockets in a frontend or worker process share this context. This setting does not change the Tokio runtime size or the number of KV indexer threads. Set a positive integer before starting the process. Dynamo reads the value when it first creates the shared context; invalid values fail context initialization. Restart the process to change it. Set `1` to restore the previous single-thread setting. This variable does not affect NATS or ZMQ contexts created outside the shared event-plane transport. --- **`--connector`** `string` — default: null KV-cache transfer connector. Accepts zero or more values. Deprecated for vLLM — use `--kv-transfer-config` instead. For TRT-LLM, valid options are `nixl`, `lmcache`, `kvbm`, `null`, and `none`. Environment variable: `DYN_CONNECTOR` --- ## Parsing **`--dyn-tool-call-parser`** `string` — default: null Tool call parser name for the model. When unset, the backend's default tool call parsing is used. Valid parser names are determined at runtime by the installed `dynamo._core` extension — run `python -m dynamo.vllm --help` to see the available choices on your installation. Environment variable: `DYN_TOOL_CALL_PARSER` --- **`--dyn-reasoning-parser`** `string` — default: null Reasoning/chain-of-thought parser name for the model. When unset, no reasoning parsing is performed. Valid parser names are determined at runtime by the installed `dynamo._core` extension — run `python -m dynamo.vllm --help` to see the available choices on your installation. Environment variable: `DYN_REASONING_PARSER` --- **`--dyn-default-thinking-mode`** `string` — default: null Thinking mode to apply when a request omits an explicit thinking control. Request-level `thinking`, `reasoning_effort`, and controls in `chat_template_args` or `chat_template_kwargs` take precedence. When unset, Dynamo preserves the model or chat template's native default. See [Deployment-Level Thinking Default](/dynamo/dev/parsing/reasoning-parsing#deployment-level-thinking-default) for examples and precedence details. Allowed values: enabled disabled Environment variable: `DYN_DEFAULT_THINKING_MODE` --- **`--exclude-tools-when-tool-choice-none`** `boolean` — default: true Exclude tool definitions from the chat template when `tool_choice='none'`. Prevents models from generating unsolicited raw XML tool calls in the content field. This flag controls the Rust-native chat template path; a matching flag in `FrontendArgGroup` controls the Python processor side independently. Environment variable: `DYN_EXCLUDE_TOOLS_WHEN_TOOL_CHOICE_NONE` --- **`--dyn-structural-tag`** `boolean or JSON object` — default: enabled Structural tags default to enabled with scope `always` and schema mode `auto` for regular workers. Pass `false` to disable optional guidance, or provide a JSON object for advanced options. Rust retains native tags for Kimi K2 required/named and Kimi K3 named choices; Python vLLM and SGLang respect the opt-out. Native Rust sidecars retain their conservative default. See [Structural Tag](/dynamo/dev/parsing/structural-tag) for configuration and backend compatibility limitations. Environment variable: `DYN_STRUCTURAL_TAG` --- **`--custom-jinja-template`** `string` — default: null Path to a custom Jinja template file to override the model's default chat template. This template takes precedence over any template found in the model repository. Environment variable: `DYN_CUSTOM_JINJA_TEMPLATE` --- ## Multimodal and media output **`--multimodal-embedding-cache-capacity-gb`** `number` — default: 0 Capacity of the multimodal embedding cache in GB. Set to `0` to disable the cache entirely. Increase this for workloads with repeated image or audio inputs to avoid redundant re-encoding. Environment variable: `DYN_MULTIMODAL_EMBEDDING_CACHE_CAPACITY_GB` --- **`--output-modalities`** `string` — default: text Space-separated list of output modalities for omni or diffusion mode, for example `--output-modalities text image`. Defaults to `text` only. Use `image`, `video`, or `audio` when deploying image/video/audio generation models. Allowed values: text image video audio Environment variable: `DYN_OUTPUT_MODALITIES` --- **`--media-output-fs-url`** `string` — default: file:///tmp/dynamo\_media Filesystem URL for storing generated images and videos. Accepts a local `file://` path or a remote object-storage URL such as `s3://bucket/path`. Environment variable: `DYN_MEDIA_OUTPUT_FS_URL` --- **`--media-output-http-url`** `string` — default: null Base HTTP URL for rewriting media file paths in API responses, for example `http://localhost:8000/media`. When unset, raw filesystem paths are returned in the response body. Environment variable: `DYN_MEDIA_OUTPUT_HTTP_URL` --- ## Fault tolerance **`DYN_RUNTIME_INHIBITED_DURATION_SECS`** `integer` — default: 5 Maximum number of seconds that a runtime client locally removes a worker from normal routing after a request-path failure while service discovery propagates the worker state. Set to `0` to disable local inhibition. The process reads this value once when the first client initializes; restart the process after changing it. Discovery updates remain authoritative, and direct dispatch bypasses the local inhibited set while the selected worker remains in discovery. --- **`DYN_RESPONSE_STREAM_ESTABLISH_TIMEOUT_SECS`** `integer` — default: 0 Seconds a frontend waits for a worker to establish its response stream after the worker acknowledges a unary request. The timer starts at the acknowledgement, so waiting for local admission, connecting, and sending does not count. It covers worker queueing and setup before the response stream opens, and a worker that accepts requests but cannot connect back. Set it above the worst-case time a healthy worker takes to start a response. On expiry the request fails with a response timeout, or migrates when migration is enabled, and the worker is locally inhibited for `DYN_RUNTIME_INHIBITED_DURATION_SECS`. Set to `0` or leave unset to disable. Routers without fault detection and bidirectional requests do not apply this timeout. --- ## Operations **`--dump-config-to`** `string` — default: null Dump the fully resolved configuration to the specified file path at startup. Useful for auditing the effective values after all env-var and CLI overrides have been applied. Environment variable: `DYN_DUMP_CONFIG_TO` --- **`--health-check-payload`** `string` — default: null Override the runtime health-check canary payload used by the unified backend's `Worker`. Accepts a JSON object string, for example `'{"token_ids": [1], "stop_conditions": {"max_tokens": 1}}'`, or a file reference prefixed with `@`, for example `@/path/to/payload.json`. Takes precedence over the engine's default `health_check_payload()`. Applies to the unified backend only. Environment variable: `DYN_HEALTH_CHECK_PAYLOAD` --- **`--engine-request-limit`** `integer` — default: null Maximum requests handled concurrently by the engine. Setting a positive integer enables worker-side admission control. When the engine slots and the Dynamo overflow queue are full, the worker rejects the request and the Frontend returns the configured overload status, 529 by default. Environment variable: `DYN_ENGINE_REQUEST_LIMIT` --- **`DYN_DYNAMO_REQUEST_QUEUE_LIMIT`** `integer` — default: 16 Advanced overflow-queue size for requests waiting in Dynamo before entering the engine. Applies only when `--engine-request-limit` or `DYN_ENGINE_REQUEST_LIMIT` is set. Must be at least `2`; the effective per-worker cap is the engine limit plus this queue limit. --- **`DYN_SYSTEM_HOST`** `string` — default: 0.0.0.0 Listen address for the system status server on `DYN_SYSTEM_PORT`, which serves health, metrics, and self-hosted model metadata. `0.0.0.0` accepts IPv4 connections only; on IPv6-only hosts, set `::`. For a wildcard address, Dynamo advertises a non-loopback address from the same family; `::` also accepts IPv4 connections, so it falls back to a non-loopback IPv4 address. If none exists, Dynamo advertises loopback and logs a warning; remote callers such as the Frontend cannot reach that address. --- ## Related pages #### [vLLM Configuration](/dynamo/dev/reference/backends/v-llm-configuration) Dynamo wrapper flag reference for the vLLM backend (`DYN_VLLM_*`). #### [SGLang Configuration](/dynamo/dev/reference/backends/sg-lang-configuration) Dynamo wrapper flag reference for the SGLang backend (`DYN_SGL_*`). #### [TensorRT-LLM Configuration](/dynamo/dev/reference/backends/tensor-rt-llm-configuration) Dynamo wrapper flag reference for the TensorRT-LLM backend. #### [Frontend Configuration](/dynamo/dev/reference/components/frontend-configuration) Configuration reference for the Dynamo HTTP frontend component.