Kimi-K3.1-Pro

Serve moonshotai/Kimi-K3.1-Pro with NVIDIA Dynamo using vLLM or SGLang. Pick your framework, GPU, and serving topology; deployment commands update to match. SGLang targets are experimental, text-only, and functional-only; no performance claims are made.

Prerequisites

• A Hugging Face token with access to moonshotai/Kimi-K3.1-Pro.
• Access to the runtime images referenced by the selected manifest.

Create the namespace and token secret:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

Deploy

kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s

Deploy the selected DynamoGraphDeployment (DGD):

export DGD=kimi-k3-1-pro-vllm-gb300-agg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-gb300-disagg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-sglang-gb300-agg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-sglang-gb300-disagg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/sglang/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-b200-agg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-vr200-agg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-vr200-disagg-agentic
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

Each target’s Kustomize overlay produces the corresponding manifest above. Customize node placement and registry credentials for your cluster before deployment; the published manifests do not include node selectors, tolerations, or image pull secrets.

B200 and VR200 targets enable DSpark with natural acceptance. Their manifests include a commented synthetic-acceptance alternative for benchmarking.

Smoke Test

The deploy command sets DGD to the selected resource name. Wait for readiness and forward the frontend port:

kubectl wait --for=condition=Ready dynamographdeployment/${DGD} \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}

Send a chat completion with the served model name moonshotai/Kimi-K3.1-Pro:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"moonshotai/Kimi-K3.1-Pro","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":256}'

Benchmark

Use perf.yaml to run AIPerf 0.13.0 against the deployment. It runs aiperf profile with the chat endpoint type and streaming on the AgentX agentic trace (semianalysis_cc_traces_weka_062126), scenario inferencex-agentx-mvp, with an agentic warmup and a 1200 s profiling phase. The tokenizer is resolved by Hugging Face ID with HF_HOME=/shared-model-cache.

The performance results use text-only AgentX traffic. For B200 and VR200 benchmarking, edit the selected target’s kustomize/base/deploy.yaml: uncomment both lines of the synthetic SPECULATIVE_CONFIG alternative and comment out both lines of the active natural-acceptance entry. Keep exactly one active entry per worker; update both prefill and decode for disaggregated VR200. Regenerate and apply the selected manifest before running AIPerf:

python3 scripts/kustomize-matrix.py render recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/.kustomize-matrix.yaml
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml -n ${NAMESPACE}

python3 scripts/kustomize-matrix.py render recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/.kustomize-matrix.yaml
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

python3 scripts/kustomize-matrix.py render recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/.kustomize-matrix.yaml
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

Wait for the replacement workers to become Ready before measuring. Restore natural acceptance, regenerate, and reapply the manifest before returning to normal serving.

The synthetic acceptance length of 3.84 was measured offline on the SpeedBench (qualitative) coding subset using the same DSpark draft length as this recipe: 7 draft tokens. The AgentX dataset and scenario above describe the throughput benchmark, not the offline acceptance-length measurement.

Set these values in the Job env section for your target:

• ENDPOINT: the frontend Service of the DynamoGraphDeployment, ${DGD}-frontend:8000 (substitute the selected deployment name).
• CONCURRENCY: GB300 uses 32 for vLLM aggregated and 48 for disaggregated. Reported VR200 points use 180 for aggregated and 174 for disaggregated; B200 uses 48.

The benchmark Job uses the same shared-model-cache PVC and tokenizer files as the deployment.

kubectl apply -f recipes/kimi-k3.1-pro/perf/perf.yaml -n ${NAMESPACE}
kubectl logs -f job/kimi-k3-1-aiperf -n ${NAMESPACE}

Results are written to /shared-model-cache/perf/<epoch>_<job-name>/. Restart the workers between independent runs to reset KV-cache and routing state.

Optimization Targets

The AgentX benchmark targets use the following interactivity goals:

Performance Results

All results below measure text-only traffic; the selected target is highlighted. B200 and VR200 use synthetic draft acceptance (AL 3.84); GB300 has speculative decoding disabled.

Compare All Targets

GPU counts below describe the checked-in manifests. P counts prefill workers; D counts decode workers.

Notes

• SGLang targets are experimental, text-only, and functional-only. No SGLang performance claims are made. Use the vLLM multimodal overlays for image and video input.
• Synthetic draft acceptance is for benchmarking. Keep the natural-acceptance configuration active for normal serving.
• Disaggregated performance depends on your fabric; RDMA tuning is cluster-specific.

Source

• Deployment assets: Kimi-K3.1-Pro recipes

• Setup assets: model-cache.yaml and model-download.yaml

• Benchmark manifest: perf.yaml

• B200 aggregated: deployment manifest

• VR200 aggregated: deployment manifest

• VR200 disaggregated: deployment manifest