> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.nvidia.com/dynamo/dev/llms.txt
> - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Frontend Configuration Reference

This page documents the configuration options for the Dynamo Frontend (`python -m dynamo.frontend`).

Every CLI argument has a corresponding environment variable. The CLI argument takes precedence; the environment variable is the fallback.

> **Note**
>
> These are the frontend-specific flags. The frontend also parses the shared Dynamo runtime flags (namespace, discovery, planes, parsing) — see [Runtime Configuration](/dynamo/dev/reference/components/runtime-configuration). For the router cost model and tuning guidance behind the router flags below, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning).

## HTTP and networking

**`--http-host`** `string` — default: 0.0.0.0

HTTP listen address: an IPv4 or IPv6 address, with optional brackets around IPv6. `0.0.0.0`
accepts IPv4 connections only; on IPv6-only hosts, set `::`.

Environment variable: `DYN_HTTP_HOST`

---

**`--http-port`** `integer` — default: 8000

HTTP listen port.

Environment variable: `DYN_HTTP_PORT`

---

**`--tls-cert-path`** `string` — default: null

TLS certificate path (PEM). Must be paired with `--tls-key-path`.

Environment variable: `DYN_TLS_CERT_PATH`

---

**`--tls-key-path`** `string` — default: null

TLS private key path (PEM). Must be paired with `--tls-cert-path`.

Environment variable: `DYN_TLS_KEY_PATH`

---

**`--tls-client-ca-cert-path`** `string` — default: null

Trusted client CA certificates in PEM format. Enables mutual TLS (mTLS) for the
HTTP server: clients must present a certificate signed by a trusted CA.
Requires both `--tls-cert-path` and `--tls-key-path`.

Environment variable: `DYN_TLS_CLIENT_CA_CERT_PATH`

---

HTTP client authentication is configured independently of internal TCP and NATS
mTLS. These HTTP options do not configure either internal transport.

For certificate rotation behavior and the HTTP TLS configuration overview,
see the [TLS reference](/dynamo/dev/reference/components/tls-configuration#http-tls-and-mtls).

The Rust HTTP server also reads these environment variables, which are not exposed as CLI arguments:

**`DYN_HTTP_BODY_LIMIT_MB`** `integer` — default: 192

Maximum HTTP request body size in MB. This is the client-facing ingress cap.
After preprocessing, the frontend sends the worker-bound frame over the request
plane. On the default TCP plane that frame is independently limited by
`DYN_TCP_MAX_MESSAGE_SIZE` (32 MiB by default). Inline `data:` media is
forwarded once in `multi_modal_data`; a serialized frame that still exceeds the
TCP cap is rejected with HTTP 400 before the connection is written. Raise
`DYN_TCP_MAX_MESSAGE_SIZE` together with this value when operators need larger
inline-media requests, or keep HTTP images as `https://` URLs so only a
reference crosses the request plane.

See [Runtime Configuration](/dynamo/dev/reference/components/runtime-configuration) for `DYN_TCP_MAX_MESSAGE_SIZE`.

---

**`DYN_HTTP_LISTEN_BACKLOG`** `integer` — default: 4096

Listen backlog of the frontend socket, for both plain HTTP and TLS, capped by
the kernel at `net.core.somaxconn`. Raise it when many clients open connections
within a few seconds (for example a load client with one connection per
session): connections that arrive while the accept queue is full are delayed
or, depending on the host's TCP settings, fail.

---

**`DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS`** `integer` — default: 5

Maximum time the Frontend waits for admitted HTTP response bodies and `/v1/realtime` WebSocket
tasks to finish after shutdown begins. New inference requests are rejected while draining.

---

**`DYN_HTTP_OVERLOAD_STATUS_CODE`** `integer` — default: 529

HTTP status returned for overload and admission-control rejection. Use `503` only for clients that
cannot handle 529. Values from 200 through 999 are accepted. Informational values from 100 through
199, invalid values, and out-of-range values fall back to 529. The value is read and cached on
first use.

---

**`DYN_HTTP_SSE_KEEP_ALIVE_INTERVAL_MS`** `integer` — default: 0

Interval in milliseconds between SSE comment frames while a streaming response has no data.
Unset, `0`, invalid, or unrepresentable values disable keep-alive comments.

---

## Router

This section is the canonical CLI and environment-variable reference for the frontend's embedded
router. See [Router Guide](/dynamo/dev/knowledge-base/modular-components/router/router-guide) for deployment modes and
[Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning) for behavior and tuning guidance.

### Routing and readiness

**`--router-mode`** `string` — default: round-robin

Routing strategy. `power-of-two` samples two workers and selects the worker with fewer in-flight
requests. In disaggregated prefill mode, `power-of-two` and `least-loaded` use the synchronous
prefill fallback path.

Allowed values:

round-robin

random

power-of-two

kv

direct

least-loaded

device-aware-weighted

Environment variable: `DYN_ROUTER_MODE`

---

**`--router-min-initial-workers`** `integer` — default: 0

Minimum number of workers required before router startup continues. `0` disables the startup wait.

Environment variable: `DYN_ROUTER_MIN_INITIAL_WORKERS`

---

**`--router-session-affinity-ttl-secs`** `integer` — default: null

Enable session affinity with this router-local idle TTL in seconds. Bindings synchronize across
router replicas on a best-effort basis. Valid values are `1` through `31536000`; omit the option
to disable session affinity. See [Session affinity](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#session-affinity).

Environment variable: `DYN_ROUTER_SESSION_AFFINITY_TTL_SECS`

---

**`--decode-fallback / --no-decode-fallback`** `boolean` — default: false

Fall back to aggregated mode when prefill workers are unavailable.

Environment variable: `DYN_DECODE_FALLBACK`

---

### KV scoring and cache locality

Configure scoring through [default policy parameters](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#configure-the-default-policy) in `--router-policy-config`. Deprecated flags and their environment variables remain accepted until removal in v1.7.

**`--load-aware / --no-load-aware`** `boolean` — default: false

**Deprecated.** Use [policy parameters and explicit tracking settings](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#replace-the-load-aware-preset).

Preset for KV load-aware routing; implies `--router-mode kv`. Disables device-cache credit, KV event consumption, and remote/shared cache indexers. The worker-selection policy's `CACHE` declaration still controls local approximate indexing.

Environment variable: `DYN_ROUTER_LOAD_AWARE`

---

**`--router-kv-overlap-score-credit`** `float` — default: 1.0

**Deprecated.** Set `overlap_score_credit` in the default policy parameters.

Credit multiplier for device-local prefix overlap. The value must be finite and nonnegative. Cache setup follows the policy’s input requirements; the default policy skips cache inputs at zero overlap credit.
Values greater than `1.0` give device overlap extra credit, but adjusted prefill cost is
clamped at zero. See [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#tuning-guidelines).

Environment variable: `DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT`

---

**`--router-kv-overlap-score-credit-decay`** `float` — default: 0.0

**Deprecated.** Set `overlap_score_credit_decay` in the default policy parameters.

Decay rate for device-local overlap credit as active prefill load rises above the least-loaded
eligible worker. `0` disables decay; `1` halves the credit at one request-equivalent of excess
active prefill load.

Environment variable: `DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY`

---

**`--router-prefill-load-scale`** `float` — default: 1.0

**Deprecated.** Set `prefill_load_scale` in the default policy parameters.

Scale applied to adjusted prompt-side prefill load after overlap and lower-tier cache-hit credits
are subtracted. The minimum is `0.0`; there is no hard maximum.

Environment variable: `DYN_ROUTER_PREFILL_LOAD_SCALE`

---

**`--router-host-cache-hit-weight`** `float` — default: 0.75

Credit multiplier from `0.0` through `1.0` for host-pinned, CPU-tier prefix-cache hits.

Environment variable: `DYN_ROUTER_HOST_CACHE_HIT_WEIGHT`

---

**`--router-disk-cache-hit-weight`** `float` — default: 0.25

Credit multiplier from `0.0` through `1.0` for disk or other lower-tier prefix-cache hits.

Environment variable: `DYN_ROUTER_DISK_CACHE_HIT_WEIGHT`

---

**`--shared-cache-multiplier`** `float`

**Deprecated.** Set `shared_cache_multiplier` in the default policy parameters. The policy defaults to `0.5` when shared cache is enabled.

**Experimental.** Credit multiplier from `0.0` through `1.0` for external shared-cache hits.

Environment variable: `DYN_SHARED_CACHE_MULTIPLIER`

---

**`--shared-cache-type`** `string` — default: none

**Experimental.** External shared KV-cache implementation.

Allowed values:

none

hicache

Environment variable: `DYN_SHARED_CACHE_TYPE`

---

**`--router-temperature`** `float` — default: 0.0

**Deprecated.** Set `router_temperature` in the default policy parameters.

Softmax temperature for normalized worker sampling. `0` selects the lowest-cost worker
deterministically; higher values add randomness.

Environment variable: `DYN_ROUTER_TEMPERATURE`

---

### KV state and indexers

**`--router-kv-events / --no-router-kv-events`** `boolean` — default: true

Consume KV cache state events from workers. Disable this option to predict cache state from routing
decisions instead.

Environment variable: `DYN_ROUTER_USE_KV_EVENTS`

---

**`--router-ttl-secs`** `float` — default: 120.0

Block TTL in seconds for prediction-based routing. Used only with `--no-router-kv-events`.

Environment variable: `DYN_ROUTER_TTL_SECS`

---

**`--router-predicted-ttl-secs`** `float` — default: null

Enable a local predict-on-route side indexer with this TTL in seconds. This option requires KV
events and is independent of `--router-ttl-secs`, which configures pure approximate mode.

Environment variable: `DYN_ROUTER_PREDICTED_TTL_SECS`

---

**`--router-event-threads`** `integer` — default: 4

KV indexer worker threads. Values greater than `1` use the concurrent radix tree, including with
`--no-router-kv-events`.

Environment variable: `DYN_ROUTER_EVENT_THREADS`

---

**`--use-remote-indexer / --no-use-remote-indexer`** `boolean` — default: false

**Experimental.** Query a remote KV indexer served by a worker component instead of maintaining a
local primary indexer.

Environment variable: `DYN_USE_REMOTE_INDEXER`

---

**`--serve-indexer / --no-serve-indexer`** `boolean` — default: false

Serve this frontend's local KV indexers over the request plane. Requires `--router-mode kv` and is
mutually exclusive with `--use-remote-indexer`.

Environment variable: `DYN_SERVE_INDEXER`

---

**`--router-replica-sync / --no-router-replica-sync`** `boolean` — default: false

Enable best-effort active-sequence synchronization through the Runtime event plane.

Environment variable: `DYN_ROUTER_REPLICA_SYNC`

---

**`--router-tracking-hash`** `string` — default: public-xxh3-v1

Hash function for router-derived active-sequence identities. `keyed-xxh3-v1` is experimental and
requires both a tracking key file and key ID.

Allowed values:

public-xxh3-v1

keyed-xxh3-v1

Environment variable: `DYN_ROUTER_TRACKING_HASH`

---

**`--router-tracking-key-file`** `string` — default: null

File containing the provider tracking key. Keyed tracking requires exactly 32 raw bytes.

Environment variable: `DYN_ROUTER_TRACKING_KEY_FILE`

---

**`--router-tracking-key-id`** `string` — default: null

Nonempty provider-managed key epoch identifier required for keyed tracking.

Environment variable: `DYN_ROUTER_TRACKING_KEY_ID`

---

### Active load and queueing

**`--router-track-active-blocks / --no-router-track-active-blocks`** `boolean` — default: true

Track blocks used by in-progress requests for load balancing.

Environment variable: `DYN_ROUTER_TRACK_ACTIVE_BLOCKS`

---

**`--router-assume-kv-reuse / --no-router-assume-kv-reuse`** `boolean` — default: true

Assume KV cache reuse when tracking active blocks.

Environment variable: `DYN_ROUTER_ASSUME_KV_REUSE`

---

**`--router-track-output-blocks / --no-router-track-output-blocks`** `boolean` — default: false

Track output blocks during generation. With `nvext.agent_hints.osl`, fractional decay applies
to output blocks and the structurally exclusive prompt suffix; shared prompt blocks retain full weight.

Environment variable: `DYN_ROUTER_TRACK_OUTPUT_BLOCKS`

---

**`--router-decode-active-request-weight`** `float` — default: 0.0

**Deprecated.** Set `decode_active_request_weight` in the default policy parameters.

**Experimental.** Finite, nonnegative block-equivalent decode cost added for each active request on
a candidate worker. Tune this value only when decode step latency depends materially on active batch
size.

Environment variable: `DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT`

---

**`--router-track-prefill-tokens / --no-router-track-prefill-tokens`** `boolean` — default: true

Track prompt-side prefill tokens in worker load accounting.

Environment variable: `DYN_ROUTER_TRACK_PREFILL_TOKENS`

---

**`--router-prefill-load-model`** `string` — default: none

Prompt-side load model. `none` keeps static prompt load; `ais` decays the oldest active prefill
request using an AIS prediction. See [AIS prefill load model](#ais-prefill-load-model).

Allowed values:

none

ais

Environment variable: `DYN_ROUTER_PREFILL_LOAD_MODEL`

---

**`--router-queue-threshold`** `float` — default: null

**Deprecated; removal in v1.7.** Set `policy_classes[].prefill_busy_threshold_frac` in `--router-policy-config`.

Queue threshold fraction of prefill capacity. Setting a nonnegative numeric value enables queueing;
priority hints affect only requests waiting in this queue.

Environment variable: `DYN_ROUTER_QUEUE_THRESHOLD`

---

**`--router-queue-policy`** `string` — default: fcfs

**Deprecated; removal in v1.7.** Set `policy_classes[].queue_policy` in `--router-policy-config`.

Queue scheduling policy. `fcfs` optimizes tail Time To First Token (TTFT); `wspt` optimizes average
TTFT.

Allowed values:

fcfs

wspt

Environment variable: `DYN_ROUTER_QUEUE_POLICY`

---

**`--router-policy-config`** `string` — default: null

Startup-only YAML for shared `router` settings, policy-class queues, and worker-selection instances. When omitted,
`--router-queue-threshold` and `--router-queue-policy` define one synthetic policy class.
Queueing remains disabled until a threshold is configured. See [Shared Router Settings](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#shared-router-settings), [Policy-class queues](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#policy-class-queues) and [Write Custom Routing Strategies](/dynamo/dev/knowledge-base/modular-components/router/write-custom-routing-strategies).

Environment variable: `DYN_ROUTER_POLICY_CONFIG`

---

**`--router-prefill-policy`** `string` — default: null

**Deprecated; removal in v1.7.** Set `worker_selection.prefill` in `--router-policy-config`.

Worker-selection instance for disaggregated prefill workers in the current router process. This
option overrides `worker_selection.prefill` from `--router-policy-config`. It is not included in
worker model cards. Set it to `default` to use Dynamo's built-in worker selector.

Environment variable: `DYN_ROUTER_PREFILL_POLICY`

---

**`--router-decode-policy`** `string` — default: null

**Deprecated; removal in v1.7.** Set `worker_selection.decode` in `--router-policy-config`.

Worker-selection instance for decode workers in the current router process. This option
overrides `worker_selection.decode` from `--router-policy-config`. It is not included in worker
model cards. Set it to `default` to use Dynamo's built-in worker selector. Configure an aggregated
pool with `worker_selection.aggregated` or `DYN_ROUTER_WORKER_SELECTION_POLICY`.

Environment variable: `DYN_ROUTER_DECODE_POLICY`

---

## AIS prefill load model

**`--ais-perf-config`** `JSON object or JSON/YAML file` — default: null

Complete AISimulate `ForwardPassPerfModelConfig`, including immutable worker role,
data roots, selection policy and nested estimator controls. Defaults to
`estimation_mode: auto` and `fallback_policy: deny`. Do not combine this input
with flat AIS identity flags. Cold regression is rejected by Router and Mocker.

---

Legacy `--aic-*` spellings remain input aliases. Use `--ais-*` in new configurations.

These options apply only when `--router-mode kv` is combined with `--router-prefill-load-model ais`.

When enabled, the frontend's embedded KV router predicts one expected prefill duration per admitted request, using the selected worker's overlap-derived cached prefix, then decays only the oldest active prefill request on each worker for prompt-side load accounting.

**`--ais-backend`** `string` — default: null

Backend family to model in AIS, for example `vllm` or `sglang`.

Environment variable: `DYN_AIS_BACKEND`

---

**`--ais-system`** `string` — default: null

AIS hardware/system identifier, for example `h200_sxm`.

Environment variable: `DYN_AIS_SYSTEM`

---

**`--ais-model-path`** `string` — default: null

Model path or model identifier used for AIS perf lookup.

Environment variable: `DYN_AIS_MODEL_PATH`

---

**`--ais-backend-version`** `string` — default: backend-specific

Pinned AIS database version. If omitted, Dynamo uses the backend default.

Environment variable: `DYN_AIS_BACKEND_VERSION`

---

**`--ais-tp-size`** `integer` — default: 1

Tensor-parallel size to model in AIS.

Environment variable: `DYN_AIS_TP_SIZE`

---

**`--ais-moe-tp-size`** `integer` — default: null

MoE tensor-parallel size for models that require AIS MoE parallelism.

Environment variable: `DYN_AIS_MOE_TP_SIZE`

---

**`--ais-moe-ep-size`** `integer` — default: null

MoE expert-parallel size for models that require AIS MoE parallelism.

Environment variable: `DYN_AIS_MOE_EP_SIZE`

---

**`--ais-attention-dp-size`** `integer` — default: null

Attention data-parallel size for models that require AIS MoE parallelism.

Environment variable: `DYN_AIS_ATTENTION_DP_SIZE`

---

For MoE models, AIS requires `ais_tp_size * ais_attention_dp_size == ais_moe_tp_size * ais_moe_ep_size`. For Kimi-style TP-only MoE runs, set `--ais-moe-tp-size` to the same value as `--ais-tp-size`, with `--ais-moe-ep-size 1` and `--ais-attention-dp-size 1`.

## Fault tolerance

**`--migration-limit`** `integer` — default: 0

Maximum request migrations per worker disconnect. `0` disables migration.

Environment variable: `DYN_MIGRATION_LIMIT`

---

**`--migration-max-seq-len`** `integer` — default: null

Maximum prompt-plus-output sequence length that remains eligible for migration. When a request
strictly exceeds the limit, migration and token tracking stop for that request. Must be from `1`
through `4294967295`. Unset means no sequence-length limit.

Environment variable: `DYN_MIGRATION_MAX_SEQ_LEN`

---

**`--active-decode-blocks-threshold`** `float` — default: null

KV cache utilization fraction from `0.0` through `1.0`. A numeric value independently enables
decode-block busy rejection. Decode-block telemetry requires `--router-mode kv`; use
`--router-track-output-blocks` when generated tokens should contribute to the observed load.

Environment variable: `DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD`

---

**`--active-prefill-tokens-threshold`** `integer` — default: null

Absolute active-prefill-token threshold. A numeric value greater than or equal to `0` independently
enables this check. Uses OR logic with the fractional prefill threshold.

Environment variable: `DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD`

---

**`--active-prefill-tokens-threshold-frac`** `float` — default: null

Non-negative fraction of `max_num_batched_tokens` used as the active-prefill-token threshold. A
numeric value independently enables this check. Uses OR logic with the absolute prefill threshold.

Environment variable: `DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD_FRAC`

---

For guidance on choosing thresholds, see [Request Rejection](/dynamo/dev/kubernetes/fault-tolerance/request-rejection) and [Request Migration](/dynamo/dev/kubernetes/fault-tolerance/request-migration).

## Model discovery

**`--namespace`** `string` — default: null

Exact namespace for model discovery scoping.

Environment variable: `DYN_NAMESPACE`

---

**`--namespace-prefix`** `string` — default: null

Namespace prefix for discovery (for example, `ns` matches `ns`, `ns-abc123`). Takes precedence over `--namespace`.

Environment variable: `DYN_NAMESPACE_PREFIX`

---

**`--model-name`** `string` — default: null

Override model name string.

Environment variable: `DYN_MODEL_NAME`

---

**`--model-path`** `string` — default: null

Path to a local model directory (for private/custom models).

Environment variable: `DYN_MODEL_PATH`

---

**`--kv-cache-block-size`** `integer` — default: null

KV cache block size override.

Environment variable: `DYN_KV_CACHE_BLOCK_SIZE`

---

## Infrastructure

**`--discovery-backend`** `string` — default: etcd

Service discovery backend.

Allowed values:

kubernetes

etcd

file

mem

Environment variable: `DYN_DISCOVERY_BACKEND`

---

**`--request-plane`** `string` — default: tcp

Request distribution transport. `tcp` is the fastest.

Allowed values:

tcp

nats

Environment variable: `DYN_REQUEST_PLANE`

---

**`--event-plane`** `string` — default: zmq

Event publishing transport. ZMQ is the default for every discovery backend, including `etcd`
and `kubernetes`, when neither `--event-plane` nor `DYN_EVENT_PLANE` is set. Set
`--event-plane nats` or `DYN_EVENT_PLANE=nats` to use NATS instead.

Allowed values:

nats

zmq

Environment variable: `DYN_EVENT_PLANE`

---

## KServe gRPC

**`--kserve-grpc-server / --no-kserve-grpc-server`** `boolean` — default: false

Start the KServe gRPC v2 server.

Environment variable: `DYN_KSERVE_GRPC_SERVER`

---

**`--grpc-metrics-port`** `integer` — default: 8788

HTTP metrics port for the gRPC service.

Environment variable: `DYN_GRPC_METRICS_PORT`

---

The gRPC server also supports optional HTTP/2 flow-control tuning via environment variables:

**`DYN_GRPC_INITIAL_CONNECTION_WINDOW_SIZE`** `integer` — default: 65536

HTTP/2 connection-level flow control window size in bytes. Default is the tonic default (64 KB).

---

**`DYN_GRPC_INITIAL_STREAM_WINDOW_SIZE`** `integer` — default: 65536

HTTP/2 per-stream flow control window size in bytes. Default is the tonic default (64 KB).

---

See the [KServe gRPC Frontend](/dynamo/dev/knowledge-base/modular-components/frontend/kserve-grpc-frontend) page for KServe message formats, backend registration, and gRPC tuning examples.

## Monitoring

**`--metrics-prefix`** `string` — default: dynamo\_frontend

Prefix for frontend Prometheus metrics.

Environment variable: `DYN_METRICS_PREFIX`

---

**`--dump-config-to`** `string` — default: null

Dump the resolved config to a file path.

Environment variable: `DYN_DUMP_CONFIG_TO`

---

### Histogram buckets

Most frontend latency and size histograms use log-spaced buckets, rounded to two significant figures, always starting with a `0` edge. Each histogram is tuned by three environment variables that share a prefix: `_MIN` and `_MAX` bound the range, and `_COUNT` is the total number of bucket edges **including** that leading `0`, so `COUNT - 1` edges are spread across the range. Every edge, including the top one, is rounded to two significant figures, so the highest exported edge is `MAX` rounded rather than `MAX` itself. These variables have no CLI equivalent, and the frontend reads them once while building its metrics, so changing one requires a restart.

| Histogram                                     | Variable prefix                 | Min     | Max        | Count | Top `le` |
| --------------------------------------------- | ------------------------------- | ------- | ---------- | ----- | -------- |
| `dynamo_frontend_request_duration_seconds`    | `DYN_METRICS_REQUEST_DURATION`  | `1.0`   | `512.0`    | `10`  | `510`    |
| `dynamo_frontend_input_sequence_tokens`       | `DYN_METRICS_INPUT_SEQUENCE`    | `50.0`  | `128000.0` | `12`  | `130000` |
| `dynamo_frontend_output_sequence_tokens`      | `DYN_METRICS_OUTPUT_SEQUENCE`   | `50.0`  | `32000.0`  | `10`  | `32000`  |
| `dynamo_frontend_time_to_first_token_seconds` | `DYN_METRICS_TTFT`              | `0.001` | `480.0`    | `18`  | `480`    |
| `dynamo_frontend_inter_token_latency_seconds` | `DYN_METRICS_ITL`               | `0.001` | `80.0`     | `20`  | `80`     |
| `dynamo_frontend_embedding_latency_seconds`   | `DYN_METRICS_EMBEDDING_LATENCY` | `0.001` | `10.0`     | `14`  | `10`     |

For example, to raise the output sequence ceiling from 32k tokens to 128k:

```bash
export DYN_METRICS_OUTPUT_SEQUENCE_MAX=131072
export DYN_METRICS_OUTPUT_SEQUENCE_COUNT=12
```

Rounding can move the top edge either way, so pick a `MAX` that is already two significant figures if you need an exact ceiling. Prometheus buckets are inclusive, so `MAX=123` exports `le=120` and only observations *greater than* 120 and up to 123 fall into `+Inf` — a value of exactly 120 still lands in the `le=120` bucket. `MAX=131072` exports `le=130000`. Two of the defaults above are affected: request duration is set to `512` but exports `510`, and input sequence is set to `128000` but exports `130000`.

`dynamo_frontend_cached_tokens` shares the input sequence buckets, so `DYN_METRICS_INPUT_SEQUENCE_*` changes both histograms.

Set `DYN_METRICS_ITL_BUCKETS` to an explicit, comma-separated list of ITL boundaries in seconds. The list must contain 1 to 512 finite, non-negative, strictly increasing values and be at most 16,384 bytes long. Boundaries are used as supplied, without logarithmic spacing or rounding.

For example, to use explicit boundaries for an SGLang deployment:

```bash
export DYN_METRICS_ITL_BUCKETS=0.002,0.004,0.006,0.008,0.010,0.015,0.020,0.025,0.030,0.035,0.040,0.060,0.080,0.100,0.200,0.400,0.600,0.800,1.000,2.000,4.000,6.000,8.000
```

The explicit list takes precedence over `DYN_METRICS_ITL_MIN`, `DYN_METRICS_ITL_MAX`, and `DYN_METRICS_ITL_COUNT`, including their deprecated names. If the list is invalid, the frontend warns and uses the default log-spaced ITL boundaries, ignoring the older settings. When `DYN_METRICS_ITL_BUCKETS` is unset, the existing log-spaced configuration and defaults remain unchanged, including the 80-second upper boundary.

Matching frontend and backend bucket boundaries reduces differences in quantile interpolation. It does not make their ITL measurement boundaries or observed samples identical.

A histogram's configuration is validated as a set: `MIN` must be above zero, below `MAX`, and `COUNT` must be between 1 and 512. If any part fails, the frontend warns and reverts **all three** values for that histogram together — setting `DYN_METRICS_ITL_MIN=100` against the default `MAX` of `80.0` also discards a perfectly valid `DYN_METRICS_ITL_COUNT=20`.

A value that cannot be parsed at all is handled per variable, not as a set: it warns, that one variable falls back to its default, and the others still apply.

Use `COUNT` of at least 2. `COUNT=1` passes validation but produces the leading `0` edge and nothing else, which ignores `MIN` and `MAX` and puts every *positive* observation in `+Inf`; only a zero-valued observation is counted, in the `le=0` bucket. Because rounding to two significant figures can collapse neighboring edges, a high `COUNT` over a narrow range also yields fewer buckets than requested; the frontend warns when that removes more than a tenth of them.

Bucket edges appear directly in the `le` label, which is the only reliable way to confirm a setting took effect. These histograms are labeled by `model`, so a series exists only after a request has populated it — on a freshly restarted frontend the command below prints nothing, which means "no data yet", not "the setting was ignored". Send a request first:

```bash
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' > /dev/null

curl -s localhost:8000/metrics | grep dynamo_frontend_inter_token_latency_seconds_bucket
```

> **Warning**
>
> Changing any of these values makes new data non-comparable with data already scraped. `histogram_quantile` still returns a number when a query spans two different `le` layouts, but aggregating incompatible bucket layouts yields misleading quantiles rather than an error, so dashboards show a discontinuity at the restart that is easy to misread as a real latency change. Bucket count also drives cardinality: each edge is a separate series per value of the `model` label.

> **Note**
>
> From v0.8.0 until this release these variables were read under a doubled prefix, as in `DYN_HISTOGRAM_DYN_METRICS_ITL_MAX`, so the names in the table above had no effect on those versions. The doubled names still work but now log a deprecation warning, and support for them will be removed in a future release. Rename them to the forms above. Configurations written against v0.7.1 or earlier work again without change.

## Tokenizer

**`--tokenizer`** `string` — default: default

Tokenizer implementation. `default` uses HuggingFace; `fastokens` uses the high-performance hybrid encoder; and `basetenkenizer` uses Baseten Tokenizer for native encoding and decoding. See [Tokenizer](/dynamo/dev/knowledge-base/modular-components/frontend/tokenizer).

Allowed values:

default

fastokens

basetenkenizer

Environment variable: `DYN_TOKENIZER`

---

**`--tokenizer-fallback / --no-tokenizer-fallback`** `boolean` — default: true

Fall back to HuggingFace when `fastokens` or `basetenkenizer` cannot load the model tokenizer. Use
`--no-tokenizer-fallback` to fail model initialization instead. In dynamic mode, discovery retries
the load while the frontend continues running.

> **Warning**
>
> Automatic tokenizer fallback is deprecated and will be disabled by default in a future release.
> Set `--no-tokenizer-fallback` to adopt the future behavior now.

Environment variable: `DYN_TOKENIZER_FALLBACK`

---

## Experimental

### Python route extensions

**`--frontend-route-extension`** `string` — default: \[]

Load a trusted HTTP route provider by a name registered in the `dynamo.frontend.routes` entry-point
group or by an importable `module:function` path. Repeat the option to load multiple providers.
Providers can add static `GET` routes but cannot override built-in routes; invalid or duplicate
routes fail startup. Extensions apply only to the HTTP frontend. See
[Python Route Extensions](/dynamo/dev/knowledge-base/modular-components/frontend/python-route-extensions) for a packaged and direct-load example.

Environment variable: `DYN_FRONTEND_ROUTE_EXTENSIONS` accepts whitespace-separated values.

---

#### Handler contract

* **Provider resolution:** A registered `dynamo.frontend.routes` entry-point name takes precedence.
  If no registered name matches, a value containing `:` is resolved as `module:function`. Unknown
  names fail startup and report the available registered extensions.
* **Provider return:** The provider callable returns one `FrontendRoute` or an iterable of routes.
* **Route shape:** Extensions support static-path `GET` routes only. Path parameters, wildcards,
  other HTTP methods, asynchronous handlers, and duplicates of built-in routes are rejected.
* **Handler signature:** A synchronous `handler(ctx: FrontendExtensionContext)` returns a
  JSON-serializable body for HTTP 200, or `FrontendResponse(status_code, body)` to override the
  status code.
* **Live state:** `FrontendExtensionContext` exposes `is_ready()`, `is_cancelled()`,
  `has_any_ready_model()`, `is_model_ready_to_serve(name)`, `model_display_names()`, and
  `serving_ready_display_names()`.
* **Execution limits:** Handlers run in a small dedicated thread pool. A handler that exceeds 30
  seconds returns 503. A saturated pool rejects new extension requests with the configured overload
  status code, 529 by default.
* **Multiple providers:** Repeating the CLI option or listing whitespace-separated environment
  values loads multiple providers. Duplicate provider names are de-duplicated.

### Other experimental options

**`--enable-anthropic-api`** `boolean` — default: false

Enable `/v1/messages` (the Anthropic Messages API).

Environment variable: `DYN_ENABLE_ANTHROPIC_API`

---

**`--dyn-chat-processor`** `string` — default: dynamo

Chat processor. See [Chat Processors](/dynamo/dev/parsing/chat-processors) for how this combines with the parser flags.

Allowed values:

dynamo

vllm

sglang

Environment variable: `DYN_CHAT_PROCESSOR`

---

**`--dyn-debug-perf`** `boolean` — default: false

Log per-function timing for preprocessing (vllm processor only).

Environment variable: `DYN_DEBUG_PERF`

---

**`--dyn-preprocess-workers`** `integer` — default: auto for SGLang; 0 otherwise

Experimental. For the SGLang processor, the default is `min(2, max(0, available CPUs - 1))`,
where available CPUs account for process affinity and visible cgroup CPU quotas. The automatic
count applies to each model's preprocessing pool, just like an explicit value. With multiple
models, the total number of workers grows with the model count and can exceed the available CPU
count. Set `0` to run preprocessing on the main event loop. Nonzero values are supported only
with the SGLang processor.

Environment variable: `DYN_PREPROCESS_WORKERS`

---

**`-i / --interactive`** `boolean` — default: false

Interactive text chat mode.

Environment variable: `DYN_INTERACTIVE`

---

## Host memory allocator

The frontend performs per-request host-memory allocations. Under high CPU load, allocator
contention can cap frontend throughput before the GPU workers saturate. The frontend does not
preload jemalloc by default.

> **Note**
>
> Whether the container image ships jemalloc depends on how the image is built. Images built with
> `dynamo_runtime.Dockerfile` without an inference backend (`--framework dynamo`, the `dev` /
> `local-dev` / `runtime` targets used for development containers and backend-less components)
> include `libjemalloc2` since v1.3.0. The released vLLM, SGLang, and TensorRT-LLM images (runtime and
> dev targets), as well as the standalone frontend entrypoint image do not. Verify with
> `dpkg -L libjemalloc2 | grep 'libjemalloc.so'`; if the library is missing, either build a custom
> frontend image that includes it, or install it from an initContainer into a shared volume mounted
> by both containers and point `LD_PRELOAD` at the copied library path. Pointing `LD_PRELOAD` at a
> path that does not exist only prints a loader warning, and the process continues with the default
> glibc allocator.

> **Note**
>
> Released vLLM, SGLang, TensorRT-LLM, and standalone frontend images up to and including v1.5.0 do
> not ship `libjemalloc2`; images from later releases include it. Verify with
> `dpkg -L libjemalloc2 | grep 'libjemalloc.so'`. On older images, install the package into a shared
> volume with an initContainer or use a custom image.

**`LD_PRELOAD`** `string` — default: unset

Preload jemalloc for the frontend process. The library path depends on the container architecture:

```bash
# x86_64
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

# arm64
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2
```

In Kubernetes, set this variable on the frontend container rather than modifying the image. Locate
the installed path with `dpkg -L libjemalloc2 | grep 'libjemalloc.so'` if the image uses a different
layout.

---

> **Warning**
>
> The TensorRT-LLM image bakes its own `LD_PRELOAD` (`/opt/dynamo/libstdc++.so.6` and TensorRT-LLM's
> bundled `libnixl.so`). Assigning `LD_PRELOAD` as shown above replaces that value. On TensorRT-LLM images, prepend the jemalloc
> library instead:
>
> ```bash
> export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
> ```

**`MALLOC_CONF`** `string` — default: jemalloc defaults

Configure jemalloc when it is preloaded. The best values depend on the request workload and CPU
allocation. This is an empirical starting point, not a production default:

```bash
export MALLOC_CONF="narenas:32,tcache:true,lg_tcache_max:15,dirty_decay_ms:5000,muzzy_decay_ms:5000"
```

Validate allocator changes with the same AIPerf workload before and after enabling them. See
[Bound frontend host memory growth](/dynamo/dev/kubernetes/operations/performance-tuning#bound-frontend-host-memory-growth)
for the symptom this addresses and how to measure the effect.

---

## HTTP endpoints

The frontend exposes the following HTTP endpoints.

### OpenAI-compatible

| Method | Path                            | Description                                                            |
| ------ | ------------------------------- | ---------------------------------------------------------------------- |
| `POST` | `/v1/chat/completions`          | Chat completions (streaming and non-streaming)                         |
| `POST` | `/v1/completions`               | Text completions                                                       |
| `POST` | `/v1/embeddings`                | Text embeddings                                                        |
| `POST` | `/v1/responses`                 | Responses API                                                          |
| `POST` | `/v1/responses/input_tokens`    | Estimated input token count for the Responses API                      |
| `POST` | `/v1/images/generations`        | Image generation                                                       |
| `POST` | `/v1/images/edits`              | Image editing; see [Image API Compatibility](#image-api-compatibility) |
| `POST` | `/v1/videos/generations`        | Video generation                                                       |
| `POST` | `/v1/videos/generations/stream` | Video generation (streaming)                                           |
| `GET`  | `/v1/models`                    | List available models                                                  |

### Image API Compatibility

The built-in image routes use JSON requests. Route availability does not imply that every
backend or model supports reference-image conditioning.

| Endpoint                      | Request contract                                                                            | Support                                                                                                                                                                   |
| ----------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /v1/images/generations` | JSON with a text `prompt`; optional `input_reference` for a reference-capable backend/model | Supported                                                                                                                                                                 |
| `POST /v1/images/edits`       | JSON with a `prompt` and required `input_reference`                                         | Requires a reference-capable backend/model; see the version requirements in the image-edit example                                                                        |
| `POST /v1/images/variations`  | Legacy image-variation API                                                                  | Unsupported (HTTP 404) and [deprecated by OpenAI](https://developers.openai.com/api/reference/resources/images/methods/create_variation); use `/v1/images/edits` instead. |

With vLLM-Omni, `input_reference` is a single HTTP(S) image URL or a base64 data URL such as
`data:image/png;base64,...`. It is not an uploaded file or a local filesystem path. The
frontend rejects multipart image edits with HTTP 415; direct uploads are tracked in
[#15818](https://github.com/ai-dynamo/dynamo/issues/15818).

See [Create a Variation with Image Edits](/dynamo/dev/diffusion/text-to-image#create-a-variation-with-image-edits)
for a reference-image-plus-prompt example.

### Anthropic (Experimental)

| Method | Path                        | Description                                                |
| ------ | --------------------------- | ---------------------------------------------------------- |
| `POST` | `/v1/messages`              | Anthropic Messages API (requires `--enable-anthropic-api`) |
| `POST` | `/v1/messages/count_tokens` | Token counting for the Anthropic API                       |

### Infrastructure

| Method | Path              | Description                                                                                                                 |
| ------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `GET`  | `/health`         | Health check                                                                                                                |
| `GET`  | `/live`           | Liveness check                                                                                                              |
| `GET`  | `/metrics`        | Prometheus metrics                                                                                                          |
| `GET`  | `/openapi.json`   | OpenAPI specification                                                                                                       |
| `GET`  | `/docs`           | Swagger UI                                                                                                                  |
| `POST` | `/busy_threshold` | Set independently enabled busy thresholds for a discovered model (disabled when `DYN_DISABLE_FRONTEND_ADMIN_API` is truthy) |
| `GET`  | `/busy_threshold` | Get stored busy thresholds (disabled when `DYN_DISABLE_FRONTEND_ADMIN_API` is truthy)                                       |

`POST /busy_threshold` accepts `model` plus any of `active_decode_blocks_threshold`,
`active_prefill_tokens_threshold`, and `active_prefill_tokens_threshold_frac`. A numeric field enables
only that check. The router reevaluates workers on the next worker-load or runtime-configuration
update; the call does not synchronously recompute the busy set and does not change startup-only router
options such as `--router-mode kv` or `--router-track-output-blocks`.

### Frontend feature switches

Environment variables controlling frontend extensions and protocol admission. The variables below
are disabled by default. Truthy values are `1`, `true`, `yes`, or `on`, case-insensitive.

**`DYN_IGNORE_OPENAI_FE_UNSUPPORTED_FIELDS`** `boolean` — default: false

When truthy, `/v1/chat/completions` and `/v1/completions` ignore and drop unknown top-level
request fields instead of rejecting the request. This does not forward arbitrary fields to a
backend. Named passthrough fields remain subject to their normal validation and handling.

---

**`DYN_DISABLE_FRONTEND_NVEXT`** `boolean` — default: false

When truthy, the frontend drops all request NvExt fields except `cache_salt` on
`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, and `/v1/messages`; ignores the
`x-dynamo-worker-instance-id`, `x-dynamo-prefill-instance-id`, `x-dynamo-dp-rank`,
`x-dynamo-prefill-dp-rank`, `x-dynamo-request-priority`, and
`x-dynamo-request-strict-priority` routing headers and their compatibility aliases; and ignores
the response-side `nvext.extra_fields` opt-in. Cache isolation remains active. The frontend
continues to use non-empty `nvext.cache_salt` and `x-tenant-id` values on these four endpoint
groups. Top-level cache salts remain active on chat completions, completions, classify, and
pooling. On embeddings, classify, and pooling, the switch only drops legacy NvExt annotations;
these endpoints do not use `x-tenant-id` or `x-dynamo-*` routing headers in either mode.

---

**`DYN_DISABLE_FRONTEND_ADMIN_API`** `boolean` — default: false

When truthy, `GET /busy_threshold` and `POST /busy_threshold` are not registered and return 404.
Inference, metrics, models, health, and liveness routes are unaffected.

---

### Endpoint path customization

All endpoint paths can be overridden via environment variables:

| Env Var                           | Default Path           |
| --------------------------------- | ---------------------- |
| `DYN_HTTP_SVC_CHAT_PATH_ENV`      | `/v1/chat/completions` |
| `DYN_HTTP_SVC_CMP_PATH_ENV`       | `/v1/completions`      |
| `DYN_HTTP_SVC_EMB_PATH_ENV`       | `/v1/embeddings`       |
| `DYN_HTTP_SVC_RESPONSES_PATH_ENV` | `/v1/responses`        |
| `DYN_HTTP_SVC_MODELS_PATH_ENV`    | `/v1/models`           |
| `DYN_HTTP_SVC_ANTHROPIC_PATH_ENV` | `/v1/messages`         |
| `DYN_HTTP_SVC_HEALTH_PATH_ENV`    | `/health`              |
| `DYN_HTTP_SVC_LIVE_PATH_ENV`      | `/live`                |
| `DYN_HTTP_SVC_METRICS_PATH_ENV`   | `/metrics`             |

## Deprecated

**`--admission-control`** `string` — deprecated, default: null

**Deprecated and ignored.** Configure the three busy thresholds directly. The compatibility flag
and `DYN_ADMISSION_CONTROL` are accepted only so older launch commands continue to start.

Environment variable: `DYN_ADMISSION_CONTROL`

---

## Related pages

#### [Runtime Configuration](/dynamo/dev/reference/components/runtime-configuration)

Shared Dynamo runtime flags parsed by the frontend and every backend.

#### [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning)

Router cost model and tuning guidance behind the router flags on this page.

#### [KServe gRPC Frontend](/dynamo/dev/knowledge-base/modular-components/frontend/kserve-grpc-frontend)

KServe gRPC configuration and integration details.

#### [NVIDIA Request Extensions (nvext)](/dynamo/dev/additional-resources/nvidia-request-extensions-nvext)

Custom per-request fields for routing, preprocessing, and response metadata.