Installation

Install Gateway API Routing

Install Gateway API, the Inference Extension, and a compatible Gateway implementation for Dynamo EPP routing.
View as Markdown

Complete this installation only if you plan to use the Gateway API routing topology. The default Dynamo Frontend topology works without Gateway API, the Gateway API Inference Extension (GAIE), or an Endpoint Picker Plugin (EPP). To use the default topology, continue to KV-Aware Routing with the Dynamo Frontend.

Install this integration after the Dynamo platform. The Dynamo operator needs the GAIE InferencePool API before it can reconcile a DynamoGraphDeployment with an EPP component.

1

Choose a Gateway implementation

Dynamo requires a Gateway implementation that supports GAIE InferencePool resources and calls the configured EPP before forwarding a request.

  • Use agentgateway when the cluster does not already standardize on Istio or when you want a focused AI gateway deployment.
  • Use Istio when the cluster already relies on Istio for ingress, service-mesh policy, or telemetry.

Other Gateway implementations can work when they support the same GAIE endpoint-picker contract.

2

Set the installation variables

Set NAMESPACE for the model, EPP, InferencePool, and HTTPRoute. With the agentgateway install method the Gateway itself lives in AGW_NAMESPACE alongside its controller; the Istio method creates the Gateway in NAMESPACE. Keep the Dynamo version aligned with the platform installation.

$export NAMESPACE=my-model
$export DYNAMO_SYSTEM_NAMESPACE=dynamo-system
$export DYNAMO_VERSION=1.3.0
$export AGW_NAMESPACE=agentgateway-system
3

Install the Gateway API layer

From the root of a Dynamo source checkout, run the repository installation script:

$./deploy/inference-gateway/scripts/install_gaie_crd_agentgateway.sh

The script installs:

  • Gateway API CRDs.
  • GAIE CRDs.
  • agentgateway with the inference extension enabled.
  • An AgentgatewayParameters resource that prevents Istio sidecar injection into Gateway proxy pods.
  • A Gateway named inference-gateway in $AGW_NAMESPACE (co-located with the agentgateway controller). Its http listener sets allowedRoutes.namespaces.from: All, so HTTPRoute objects in $NAMESPACE attach to it across namespaces.

The workload namespace $NAMESPACE still holds the model, EPP, InferencePool, and HTTPRoute. Only the Gateway itself lives in $AGW_NAMESPACE.

The script pins compatible dependency versions. Review it before running it when your platform team owns Gateway API or agentgateway lifecycle.

4

Verify the required APIs

Confirm that the cluster serves the Gateway, route, and inference-pool resources:

$kubectl api-resources | grep -E 'gateways|httproutes|inferencepools'
$kubectl get gatewayclass

Expected API groups include:

gateway.networking.k8s.io
inference.networking.k8s.io
5

Verify the Gateway

The Gateway namespace depends on the install method: the agentgateway script creates it in $AGW_NAMESPACE, while the Istio flow creates it in $NAMESPACE.

Wait for the inference-gateway resource to become programmed:

$kubectl wait gateway/inference-gateway -n "$AGW_NAMESPACE" \
>  --for=condition=Programmed --timeout=180s
$
$kubectl get gateway inference-gateway -n "$AGW_NAMESPACE"

If Programmed remains false, inspect the Gateway and the controller:

$kubectl describe gateway inference-gateway -n "$AGW_NAMESPACE"
$kubectl get pods --all-namespaces | grep agentgateway

Next Step

Add an EPP component and route an existing DGD with Route Requests with Gateway API.