> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Kimi-K3 Each target below is a Dynamo + vLLM or SGLang deployment of Moonshot AI's Kimi-K3, a multimodal MoE model serving up to 1M-token context, on H200, GB200, or GB300. Pick your framework, GPU architecture, and serving topology; every command on this page updates to match. Choose your deployment target Framework SGLang vLLM GPU H200 GB200 GB300 Topology Aggregated Disaggregated **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, BF16 KV **GPUs** 32x H200 (1 replica) **Parallelism** TP8, PP4 **Routing** KV-aware **Speculative decoding** None **Status** No recipe available for this combination **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 16x GB200 (1 replica) **Parallelism** TP16 **Routing** KV-aware **Speculative decoding** None **Status** No recipe available for this combination **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 16x GB300 (1 replica) **Parallelism** DCP16 attention, TEP16 MoE **Routing** KV-aware **Speculative decoding** DSPARK **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 16x GB300 (1P1D) **Parallelism** TP8 per role **Routing** KV-aware **Speculative decoding** DSPARK **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 16x GB300 (2 replicas) **Parallelism** DCP8 attention, TP8/EP1 MoE **Routing** Load-aware **Speculative decoding** DSPARK **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 32x GB200 (2 replicas) **Parallelism** DCP16 attention, TEP16 MoE **Routing** Load-aware **Speculative decoding** DSPARK **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 48x GB200 (2P1D) **Parallelism** Prefill TEP16; decode DCP16 attention, TEP16 MoE **Routing** Load-aware **Speculative decoding** DSPARK **Status** No recipe available for this combination **Status** No recipe available for this combination **Checkpoint** moonshotai/Kimi-K3 **Precision** MXFP4 experts, FP8 KV **GPUs** 16x GB300 (1P1D) **Parallelism** Prefill TEP8; decode DCP8 attention, TEP8 MoE **Routing** Load-aware **Speculative decoding** DSPARK ## Prerequisites * A Kubernetes cluster with the Dynamo platform installed and 32 H200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). * A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3`. - A Kubernetes cluster with the Dynamo platform installed and 16 GB200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). - The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). - A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3`. * A Kubernetes cluster with the Dynamo platform installed and 16 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). * The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). * A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `Inferact/Kimi-K3-DSpark`. - A Kubernetes cluster with the Dynamo platform installed and 16 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). - The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). - A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `Inferact/Kimi-K3-DSpark`. * A Kubernetes cluster with the Dynamo platform installed and 32 GB200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). * The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). * A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `RadixArk/Kimi-K3-DSpark`. - A Kubernetes cluster with the Dynamo platform installed and 48 GB200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). - The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). - A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `RadixArk/Kimi-K3-DSpark`. * A Kubernetes cluster with the Dynamo platform installed and 16 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). * The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). * A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `RadixArk/Kimi-K3-DSpark`. - A Kubernetes cluster with the Dynamo platform installed and 16 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). - The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). - A Hugging Face token with access to the checkpoint `moonshotai/Kimi-K3` and draft model `RadixArk/Kimi-K3-DSpark`. Create the namespace and token secret: ```bash export NAMESPACE=your-namespace kubectl create namespace ${NAMESPACE} kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN="your-token" \ -n ${NAMESPACE} ``` > **Warning** > > The recipes require the `shared-model-cache` PVC. Edit `storageClassName` in `model-cache/model-cache.yaml` to a ReadWriteMany storage class on your cluster (`kubectl get storageclass`) before applying it. Review namespace, image tags, and resource claims in the manifests as well. ## Deploy Create the `shared-model-cache` PVC and download the Kimi-K3 checkpoint and both DSPARK draft models. The download can take several hours on a cold cache: ```bash # Edit storageClassName in model-cache/model-cache.yaml first. kubectl apply -f recipes/kimi-k3/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/kimi-k3/model-cache/model-download.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s ``` The workers mount the PVC at `/shared-model-cache` and resolve the checkpoint and any configured DSPARK draft model from the local Hugging Face cache with `HF_HUB_OFFLINE=1`. Deploy the selected DynamoGraphDeployment (DGD): ```bash export DGD=kimi-k3-vllm-h200-agg-agentic kubectl apply -f recipes/kimi-k3/vllm/agg-h200-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-vllm-gb200-agg-agentic kubectl apply -f recipes/kimi-k3/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-vllm-gb300-agg-agentic kubectl apply -f recipes/kimi-k3/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-vllm-gb300-disagg-agentic kubectl apply -f recipes/kimi-k3/vllm/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-sglang-gb200-agg-agentic kubectl apply -f recipes/kimi-k3/sglang/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-sglang-gb200-disagg-agentic kubectl apply -f recipes/kimi-k3/sglang/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-sglang-gb300-agg-agentic kubectl apply -f recipes/kimi-k3/sglang/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash export DGD=kimi-k3-sglang-gb300-disagg-agentic kubectl apply -f recipes/kimi-k3/sglang/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` The first worker launch loads weights and captures CUDA graphs, which can take tens of minutes. ## Smoke Test Send a test request to verify the selected deployment serves traffic. The deploy command sets `DGD` to its resource name. Forward the frontend port: ```bash kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE} ``` All Kimi-K3 recipes serve under the same model name, `moonshotai/Kimi-K3`: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"moonshotai/Kimi-K3","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}' ``` Kimi-K3 reasons before answering, and tool calling is supported — the deployment uses the `kimi_k3` reasoning and tool-call parsers at the frontend. ## Benchmark See [perf/README.md](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/perf/README.md) for the full benchmark workflow: staging the trace on the PVC, running the AIPerf trace-replay Job, running a concurrency sweep, and fetching artifacts. ### Optimization Targets Recipes are optimized for the following agentic workload and target user interactivity: | Workload | Median ISL | Median OSL | KV cache hit rate | User output tok/s | TTFT P50 | | -------- | ---------: | ---------: | ----------------: | ----------------: | -------: | | Agentic | 64k | 400 | 90% | 50 | ≤ 5 s | The benchmark replays the Mooncake-format agentic trace described in [perf/README.md](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/perf/README.md). ### Performance Results Benchmarking uses a synthetic acceptance length with the SpeedBench coding AL. | Workload | Framework | Recipe | SKU | Concurrency | System output tok/s/GPU | User output tok/s (P50) | TTFT P50 (ms) | | -------------------- | --------- | ---------------------- | ----- | ----------: | ----------------------: | ----------------------: | ------------: | | Agentic (15% subset) | SGLang | Aggregated (2 workers) | GB200 | 28 | 32.9 | 56.1 | 828 | | Agentic (15% subset) | SGLang | Disaggregated (2P1D) | GB200 | 24 | 29.3 | 77.9 | 3,090 | | Agentic (15% subset) | SGLang | Aggregated (2 workers) | GB300 | 36 | 91.3 | 55.6 | 486 | | Agentic (15% subset) | SGLang | Disaggregated (1P1D) | GB300 | 18 | 62.0 | 77.6 | 4,793 | | Agentic (15% subset) | vLLM | Aggregated (1 worker) | H200 | 4 | 3.54 | 39.95 | 3660 | | Agentic (15% subset) | vLLM | Aggregated (1 worker) | GB200 | 7 | 20.1 | 56.1 | 638 | | Agentic (15% subset) | vLLM | Aggregated (1 worker) | GB300 | 24 | 62.2 | 57.8 | 879 | | Agentic (15% subset) | vLLM | Disaggregated (1P1D) | GB300 | 16 | 50.3 | 90.0 | 6,357 | ## Compare All Targets | Framework | SKU | Recipe | Precision | Routing | Speculative decoding | Context length | | --------- | ----- | ---------------------- | ---------------------- | ---------- | -------------------- | -------------- | | vLLM | GB300 | Aggregated (1 worker) | MXFP4 experts, FP8 KV | KV-aware | DSPARK | 1M | | vLLM | GB300 | Disaggregated (1P1D) | MXFP4 experts, FP8 KV | KV-aware | DSPARK | 1M | | vLLM | GB200 | Aggregated (1 worker) | MXFP4 experts, FP8 KV | KV-aware | None | 1M | | vLLM | H200 | Aggregated (1 worker) | MXFP4 experts, BF16 KV | KV-aware | None | 1M | | SGLang | GB300 | Aggregated (2 workers) | MXFP4 experts, FP8 KV | Load-aware | DSPARK | 1M | | SGLang | GB300 | Disaggregated (1P1D) | MXFP4 experts, FP8 KV | Load-aware | DSPARK | 1M | | SGLang | GB200 | Aggregated (2 workers) | MXFP4 experts, FP8 KV | Load-aware | DSPARK | 1M | | SGLang | GB200 | Disaggregated (2P1D) | MXFP4 experts, FP8 KV | Load-aware | DSPARK | 1M | ## Workaround: Health Check During Long Prefill Dynamo v0.5.1 and later run a canary health check on each worker. The check sends a small `generate` request with a timeout of 3 seconds (`DYN_HEALTH_CHECK_REQUEST_TIMEOUT`). The operator turns the check on from its runtime feature gate. During a long prefill, the check can wait for more than 3 seconds. The liveness probe then fails, and Kubernetes restarts a worker that is busy but healthy. Before the restart, the worker log shows a `Health check timeout for` warning that names the `generate` endpoint. If you see this warning, turn off the canary check on each SGLang worker: ```yaml - name: DYN_HEALTH_CHECK_ENABLED value: "false" ``` The manifest value overrides the operator default. With the canary off, Kubernetes does not restart the worker during a long prefill. The trade-off: liveness no longer checks the `generate` endpoint. ## Limitations * With JSON structured decoding, output for non-object top-level items can appear in `reasoning_content` instead of `content`. * Requests containing `logprobs` or `stop_token_ids` are not supported and may cause disaggregated recipes to crash or enter a bad state ## Source * Setup assets: [model-cache.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3/model-cache/model-cache.yaml) and [model-download.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3/model-cache/model-download.yaml) * Benchmark instructions: [recipes/kimi-k3/perf/README.md](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/perf/README.md) - Selected recipe: [vLLM aggregated GB200 deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3/vllm/agg-gb200-agentic/deploy.yaml) * Selected recipe: [vLLM aggregated GB300 deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3/vllm/agg-gb300-agentic/deploy.yaml) - Selected recipe: [vLLM disaggregated GB300 deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3/vllm/disagg-gb300-agentic/deploy.yaml) * Selected recipe: [vLLM aggregated H200 deploy.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/vllm/agg-h200-agentic/deploy.yaml) - Selected recipe: [SGLang aggregated GB200 deploy.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/sglang/agg-gb200-agentic/deploy.yaml) * Selected recipe: [SGLang disaggregated GB200 deploy.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/sglang/disagg-gb200-agentic/deploy.yaml) - Selected recipe: [SGLang aggregated GB300 deploy.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/sglang/agg-gb300-agentic/deploy.yaml) - Benchmark Job: [perf.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/perf/perf.yaml) * Selected recipe: [SGLang disaggregated GB300 deploy.yaml](https://github.com/milesial/dynamo/blob/docs/kimi-k3-framework-selector/recipes/kimi-k3/sglang/disagg-gb300-agentic/deploy.yaml)