Kubernetes User Guide

Install Dynamo and serve a model on a Kubernetes cluster.

Check supported platforms, backends, features, and dependencies.

Install the operator, custom resource definitions, and platform services.

Configure Grove or LeaderWorkerSet for workloads that span nodes.

Compare storage options for Azure Kubernetes Service.

Mount Azure Managed Lustre with the Container Storage Interface driver.

Use Amazon Elastic File System for shared model storage.

Prepare an Amazon EKS cluster for Dynamo.

Deploy Dynamo with Amazon Elastic Container Service.

Prepare an Azure Kubernetes Service cluster for Dynamo.

Use interruptible capacity for suitable workloads.

Prepare a Google Kubernetes Engine cluster for Dynamo.

Choose a deployment workflow and learn the core Kubernetes resources.

Run Dynamo beside a stock inference engine and find backend-specific deployment examples.

Define and apply a DynamoGraphDeployment.

Route requests according to cached prompt prefixes.

Scale prefill and decode workers independently.

Select parallelism and replica settings for a workload.

Install a gang scheduler for workloads that span Kubernetes nodes.

Choose storage for model weights and shared caches.

Extend KV cache capacity beyond GPU memory.

Collect metrics from the operator and serving components.

Measure throughput and latency under load.

Learn the simulation workflow and components.

Deploy live Mocker workers without GPUs.