> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Gateway API Routing Reference Use this reference for a `DynamoGraphDeployment` (DGD) that routes requests through the Gateway API Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure and topology, see [Using GAIE with Dynamo](/dynamo/dev/kubernetes/kv-aware-routing/using-gaie-with-dynamo). ## EPP Modes The native Rust EPP runs in one of two modes, selected by the `DYN_EPP_MODE` environment variable. This reference covers Dynamo mode only. The deprecated Go EPP, kept by `eppConfig` with the image pinned to 1.4, does not read `DYN_EPP_MODE` and has no standalone mode. | Concern | Dynamo mode (`dynamo`, default) | Standalone mode (`standalone`) | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Deployed by | Dynamo operator, from a DGD `epp` component | User-applied manifests | | Dynamo runtime | Required. It uses whichever discovery and event planes the platform is configured for; etcd and NATS are needed only when selected | Not used | | Worker discovery | Dynamo runtime discovery | Kubernetes pod watch driven by `InferencePool.spec.selector` | | EPP ServiceAccount and RBAC | Cluster-wide mode: the operator creates the ServiceAccount and binds it to the `dynamo-platform` chart's EPP ClusterRole. Namespace-restricted mode: the chart creates the ServiceAccount, Role, and RoleBinding | Supplied with the deployment manifests | | EPP replication | Available. With `DYN_ROUTER_REPLICA_SYNC=true`, replicas share active-sequence state over the Dynamo event plane (ZMQ by default, NATS if configured) | Available. Replicas share active-sequence state over ZMQ, discovering peers through the Service named by `DYN_EPP_PEER_SERVICE` | A standalone EPP is not managed by the Dynamo operator, so the operator generates no resources for it. It uses neither the EPP ServiceAccount nor the `dynamo-platform` chart's EPP RBAC. Standalone mode targets clusters without the Dynamo platform chart; do not install it for this path. A standalone EPP carries its own ServiceAccount and Role, which grant the permissions that mode needs. Do not bind a standalone EPP to the chart's EPP ClusterRole; its rules cover only the resources a Dynamo-mode EPP reads. For the standalone path, see [Using GAIE with vanilla vLLM](/dynamo/dev/kubernetes/kv-aware-routing/vanilla-vllm-onramp). ## DGD EPP Fields The EPP is a DGD component with `type: epp`. The Dynamo operator creates the EPP workload, Service, and `InferencePool` from these fields. The Dynamo EPP runs the full Dynamo KV-aware router natively and is configured through `DYN_*` environment variables. It implements the GAIE [Lightweight Endpoint Picker (LW-EPP)](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/README.md) `ext_proc` interface and does not use a plugin or scheduling-profile config. **`spec.components[].type`** `string` — required Set to `epp` for the Endpoint Picker Plugin component. A DGD can contain at most one EPP component. Value: epp --- **`spec.components[].eppConfig`** `object` Deprecated Go-EPP configuration. When present, keep the EPP image pinned to 1.4; upgrading only the Dynamo Operator does not require migration. To select the native Rust EPP, remove this field and switch the image to 1.5 or later in the same update. Admission rejects mixed combinations. See [Route Requests with Gateway API](/dynamo/dev/kubernetes/kv-aware-routing/using-gaie-with-dynamo). --- **`spec.components[].replicas`** `integer` — default: 1 Number of EPP pods. Increase it for endpoint-picker throughput or to survive the loss of a single EPP pod. Above `1`, the routing accuracy of the fleet becomes approximate, and replicas do not share active-load state unless `DYN_ROUTER_REPLICA_SYNC` is set. Read [EPP Replication and Routing Accuracy](#epp-replication-and-routing-accuracy) before scaling it. The deprecated Go EPP, selected by the presence of `eppConfig`, does not support replication. It must have exactly `1` replica, and admission rejects any other value. --- **`spec.components[].frontendSidecar`** `string` — required Name of the Frontend container in each routable worker pod. The named container must exist in `podTemplate.spec.containers` and run with `--router-mode direct`. --- For the complete component schema, see the [DynamoComponentDeployment Reference](/dynamo/dev/reference/api/kubernetes/dynamo-component-deployment). ## EPP Runtime Environment Set these values on the EPP component's `main` container. Shared KV router settings use the same semantics as Frontend-hosted routing; this section identifies the settings most relevant to the Gateway topology. **`DYN_MODEL_NAME`** `string` Model identifier used to load tokenizer and model configuration. Set it when discovery does not provide the model name. It must match the model served by the workers. --- **`DYN_KV_CACHE_BLOCK_SIZE`** `integer` Backend KV cache block size. Set it when discovery does not provide the value. It must match the worker backend's block size; a mismatch changes prefix block hashes and produces incorrect overlap scores. --- **`DYN_USE_KV_EVENTS`** `boolean` — default: true Enables worker-published KV event consumption in the EPP router. When `false`, the router relies on predicted local state instead of precise worker cache events. --- **`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT`** `number` — default: 1.0 Credit applied to device-local prompt-prefix overlap. Higher values prefer workers that already hold more of the prompt prefix. Set to `0` to remove device-local cache overlap from endpoint scoring. --- **`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY`** `number` — default: 0.0 Reduces overlap credit as active prefill load rises above the least-loaded eligible worker. `0` disables decay. --- **`DYN_ROUTER_PREFILL_LOAD_SCALE`** `number` — default: 1.0 Scales prompt-side prefill load after cache-hit credits are applied. --- **`DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT`** `number` — default: 0.0 Adds block-equivalent routing cost for each active request on a candidate worker. Tune this value only when decode step latency depends materially on active batch size. --- **`DYN_ROUTER_TEMPERATURE`** `number` — default: 0.0 Controls worker exploration through normalized softmax sampling. `0` selects deterministically. --- **`DYN_ROUTER_REPLICA_SYNC`** `boolean` — default: false Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event plane. Has no effect at one replica, and the deprecated Go EPP cannot be replicated. Set it to `true` when running more than one EPP replica. See [EPP Replication and Routing Accuracy](#epp-replication-and-routing-accuracy). --- **`DYN_ROUTER_TRACK_ACTIVE_BLOCKS`** `boolean` — default: true Includes blocks used by active generation in worker load accounting. --- **`DYN_ROUTER_TRACK_OUTPUT_BLOCKS`** `boolean` — default: false Predicts output blocks during generation and decays them as requests make progress. --- **`DYN_ROUTER_TRACK_PREFILL_TOKENS`** `boolean` — default: true Includes active prompt-side prefill tokens in worker load accounting. --- For the full shared configuration surface and tuning guidance, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning). ## EPP Replication and Routing Accuracy Set `DYN_ROUTER_REPLICA_SYNC` to `true` when running more than one EPP replica. The EPP embeds the same KV router as the Frontend, so replicated EPPs behave like replicated Frontends: * Prefix-cache state is the same on every replica. Each replica consumes worker KV events directly, so replication does not affect cache-overlap scoring. * Active-load state is where replicas differ. Without `DYN_ROUTER_REPLICA_SYNC`, each replica sees only the requests it routed. With it, replicas share that state on a best-effort basis and can briefly disagree, for example when two replicas pick the same worker at the same time. * Router queueing, when enabled, is per replica. A replica does not see requests queued on its peers, and its queued requests are lost if it stops. For which active-load events are synchronized and which stay local, see [Active Block Replica Synchronization](/dynamo/dev/knowledge-base/modular-components/router/router-operations#active-block-replica-synchronization). ## Generated Resource Contract For a DGD named ``, the operator generates an `InferencePool` named `-pool` in the DGD namespace. Its `endpointPickerRef` points to the generated EPP Service on gRPC port `9002`, and its target port points to worker Frontend sidecars on port `8000`. ```yaml spec: selector: matchLabels: nvidia.com/dynamo-component-class: worker nvidia.com/dynamo-namespace: endpointPickerRef: kind: Service name: port: number: 9002 targetPorts: - number: 8000 ``` Treat the generated `InferencePool`, EPP Service, and Deployment as operator-owned resources. Make persistent changes in the DGD. ### Endpoint Picker Failure Mode `InferencePool.spec.endpointPickerRef.failureMode` decides what the gateway does when the EPP does not answer, such as when its pod is restarting or has failed: * `FailClose` drops the request. The client gets an error, in practice HTTP 500. * `FailOpen` forwards the request to an endpoint the gateway picks. The request is served, but without KV-aware routing. The operator does not set this field, so the Gateway API default of `FailClose` applies. Setting it by hand on the generated `InferencePool` does not hold: the operator restores its own rendering on the next reconcile, which leaves the field unset and lets the default apply again. No DGD field selects it. A standalone EPP is unaffected. Its `InferencePool` is user-applied, so the value you write is the value you keep. The on-ramp example leaves the field out, so it also gets `FailClose`. ## HTTPRoute Contract **`spec.parentRefs[]`** `ParentReference` — required Gateway that receives model traffic. The Gateway and route can be in different namespaces only when the Gateway listener allows the route namespace. --- **`spec.rules[].backendRefs[]`** `BackendRef` — required Reference to the operator-generated `InferencePool`. --- **`group`** `string` — required Set to `inference.networking.k8s.io`. --- **`kind`** `string` — required Set to `InferencePool`. --- **`name`** `string` — required Generated pool name, normally `-pool`. --- **`port`** `integer` — required Worker Frontend target port. The operator-generated pool uses `8000`. --- ## Request Mutation Contract The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the mutated request to the selected Frontend sidecar. **`x-dynamo-worker-instance-id`** `HTTP header` Aggregated or decode worker selected for the request. --- **`x-dynamo-dp-rank`** `HTTP header` Data-parallel rank selected for the aggregated or decode worker. --- **`x-dynamo-routing-mode`** `HTTP header` Routing topology selected for the request. Values: aggregated disaggregated --- **`x-dynamo-prefill-instance-id`** `HTTP header` Prefill worker selected for a disaggregated request. --- **`x-dynamo-prefill-dp-rank`** `HTTP header` Data-parallel rank selected for the prefill worker. --- For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into `nvext.token_data` so the Frontend sidecar does not repeat tokenization. ## Service Mesh Configuration **`dynamo.serviceMesh.enabled`** `boolean` — default: false Generates service-mesh resources for EPP Services when the provider CRDs are installed. --- **`dynamo.serviceMesh.provider`** `string` — default: istio Service-mesh provider. Only Istio is supported. Value: istio --- **`dynamo.serviceMesh.istio.tlsMode`** `string` — default: SIMPLE TLS mode in generated Istio `DestinationRule` resources. --- **`dynamo.serviceMesh.istio.insecureSkipVerify`** `boolean` — default: true Skips server certificate verification for the EPP's self-signed serving certificate. --- **`dynamo.serviceMesh.istio.clientCertificate`** `string` Client certificate path for `MUTUAL` TLS mode. --- **`dynamo.serviceMesh.istio.privateKey`** `string` Client private-key path for `MUTUAL` TLS mode. --- **`dynamo.serviceMesh.istio.caCertificates`** `string` Certificate authority path for `MUTUAL` TLS mode. --- When service-mesh integration is enabled, the operator creates a `DestinationRule` for each EPP Service. If the operator does not manage the mesh integration, create an equivalent rule so the Gateway proxy can connect to EPP gRPC port `9002`. ## agentgateway and Istio Injection When namespace-level Istio injection is enabled, an injected `istio-proxy` in an agentgateway data plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses. Use an `AgentgatewayParameters` resource in the same namespace as the `Gateway`: ```yaml apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayParameters metadata: name: inference-gateway-params spec: deployment: spec: template: metadata: annotations: sidecar.istio.io/inject: "false" ``` Reference it from the Gateway: ```yaml spec: gatewayClassName: agentgateway infrastructure: parametersRef: group: agentgateway.dev kind: AgentgatewayParameters name: inference-gateway-params ``` `AgentgatewayParameters` is a local reference and must be in the Gateway namespace. Use a per-Gateway resource unless disabling injection is an intentional cluster-wide policy. ## Developer References Image build commands belong with the component source, not in this user reference. Use this source location when developing or replacing the standard EPP image: * [Dynamo EPP source](https://github.com/ai-dynamo/dynamo/tree/main/deploy/inference-gateway/ext-proc) * [GAIE Lightweight Endpoint Picker (LW-EPP) reference](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/README.md)