跳到导航
Model RecipesZ.ai

GLM-5.3-Flash

Serve GLM-5.3-Flash with Dynamo and SGLang on B200 and GB200, or vLLM on GB200 and H200.

以 Markdown 格式查看

This recipe references an upstream third-party vLLM container image. NVIDIA does not publish or distribute this image. Users should review the upstream image’s open-source license and codec terms before use or redistribution.

Deploy Z.AI’s GLM-5.3-Flash with NVIDIA Dynamo using SGLang or vLLM. The SGLang B200 and GB200 recipes use a pinned NVFP4 checkpoint, FP8 KV cache, EAGLE ReplaySSM speculative decoding, and HiCache CPU offload. Choose single-node B200 or GB200 TP4 aggregated serving, or GB200 disaggregated serving with two TP4/EP4 prefill workers plus one TP4 decode worker, KV events, and Mooncake NVLink transfer. The vLLM targets use an FP8 checkpoint with BF16 KV cache on GB200 or H200. Pick your framework, GPU architecture, and serving topology.

Choose your deployment target

Framework
GPU
Topology
Checkpoint zai-org/GLM-5.3-FlashPrecision FP8-quantized checkpoint + BF16 KV cacheGPUs 8x H200, one workerParallelism TP8Speculation MTP7Context Up to 1,048,576 tokens
Checkpoint zai-org/GLM-5.3-FlashPrecision FP8-quantized checkpoint + BF16 KV cacheGPUs 8x H200 prefill + 8x H200 decodeParallelism TP8 prefill / TP8 decodeRouting KV-aware, NIXL KV transferSpeculation MTP7 on prefill and decodeContext Up to 1,048,576 tokens
Checkpoint zai-org/GLM-5.3-FlashPrecision FP8-quantized checkpoint + BF16 KV cacheGPUs 4x GB200, one workerParallelism TP4 over MNNVLRouting KV-awareContext Up to 1,048,576 tokens
Checkpoint zai-org/GLM-5.3-FlashPrecision FP8-quantized checkpoint + BF16 KV cacheGPUs 4x GB200 prefill + 4x GB200 decodeParallelism TP4 prefill / TP4 decode over MNNVLRouting KV-aware, NIXL/UCX KV transfer over TCPContext Up to 1,048,576 tokens
Checkpoint RadixArk/GLM-5.3-Flash-NVFP4Precision NVFP4 weights + FP8 E4M3 KV cacheGPUs 4x GB200, one TP4 workerRouting Round-robinSpeculation EAGLE, 3 steps / 4 draft tokens, ReplaySSMHiCache 100 GB per TP rank; ~400 GB per worker Pod
Checkpoint RadixArk/GLM-5.3-Flash-NVFP4Precision NVFP4 weights + FP8 E4M3 KV cacheGPUs 8x GB200 prefill + 4x GB200 decodeParallelism 2x prefill TP4/EP4 + 1x decode TP4Routing KV-aware routing with worker events, Mooncake NVLink transferSpeculation EAGLE ReplaySSM on decodeHiCache 100 GB per prefill TP rank; ~400 GB per prefill Pod
Checkpoint RadixArk/GLM-5.3-Flash-NVFP4Precision NVFP4 weights + FP8 E4M3 KV cacheGPUs 4x B200 on one amd64 node, one TP4 workerRouting Round-robinSpeculation EAGLE, 3 steps / 4 draft tokens, ReplaySSMHiCache 100 GB per TP rank; ~400 GB per worker PodOverlay nscale, single-node NVLink without ComputeDomain
For B200, select Aggregated. For SGLang disaggregated serving, select GB200.
For B200, select SGLang and Aggregated. The vLLM recipes target GB200 and H200.
SGLang recipes are available for B200 aggregated serving and GB200 aggregated/disaggregated serving. Choose vLLM for H200.

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available: 4x on one ARM64 node for aggregated, or 8x across two ARM64 nodes for disaggregated. See the Kubernetes Deployment Guide.
  • The NVIDIA DRA driver with ComputeDomain support installed. Both GB200 targets require a ComputeDomain spanning the full NVL72 rack.
  • Access to zai-org/GLM-5.3-Flash and the vllm/vllm-openai:glm53-flash image.
  • A ReadWriteMany PVC named model-cache, populated with the checkpoint in the standard Hugging Face cache layout.
  • A Kubernetes cluster with the Dynamo platform installed and H200 GPUs available: 8x on one node for aggregated, or 16x across two nodes for disaggregated. See the Kubernetes Deployment Guide.
  • Eight RDMA resource claims on each GPU node for disaggregated serving. Update rdma/ib if your device plugin advertises a different resource name.
  • Access to zai-org/GLM-5.3-Flash and the vllm/vllm-openai:glm53-flash image.
  • A ReadWriteMany PVC named model-cache, populated with the checkpoint in the standard Hugging Face cache layout.
  • A Kubernetes cluster with the Dynamo operator supporting nvidia.com/v1beta1. See the Kubernetes Deployment Guide.
  • GB200 arm64 nodes with four available GPUs per worker Pod: 4 GPUs for aggregated serving or 12 GPUs for 2P1D disaggregated serving.
  • The NVIDIA DRA driver and resource.nvidia.com/v1beta1 ComputeDomain CRD. Disaggregated workers must fit in one NVLink clique; the overlay uses Pod affinity on the nvidia.com/gpu.clique node label.
  • Standalone Kustomize v5.8.1 and a checkout of this repository.
  • Access to RadixArk/GLM-5.3-Flash-NVFP4 at revision f46cf340d35a22d0d83d0c1dac8957cf2b1bcd35.
  • A ReadWriteMany PVC named shared-model-cache, accessible from all worker nodes.

The scheduling component selects arm64 nodes and nvidia.com/gpu.product=NVIDIA-GB200 for workers. It tolerates kubernetes.io/arch=arm64:NoSchedule and, on workers, nvidia.com/gpu=present:NoSchedule. Keep site-specific namespace, storage class, pull Secrets, priority class, and scheduling overrides in a private cluster overlay.

  • A Kubernetes cluster with the Dynamo operator supporting nvidia.com/v1beta1 and four available B200 GPUs on one amd64 node. See the Kubernetes Deployment Guide.
  • Standalone Kustomize v5.8.1 and a checkout of this repository.
  • A populated ReadWriteMany PVC named shared-model-cache, accessible from the worker Pod, containing RadixArk/GLM-5.3-Flash-NVFP4 at revision f46cf340d35a22d0d83d0c1dac8957cf2b1bcd35.
  • The inference priority class, or a private overlay selecting your cluster’s priority class.

The nscale overlay selects amd64 nodes and nvidia.com/gpu.product=NVIDIA-B200 for the worker. Both Pods tolerate nvidia.com/gpu=true:NoSchedule. Adapt placement and registry access to your cluster. This single-node recipe uses NVLink/NVLS with MNNVL disabled and creates no ComputeDomain or DRA channel claims.

Create the namespace and token secret:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

Match storage, scheduling, registry access, and resource claims to your cluster. For SGLang, edit the Kustomize sources or use a private cluster overlay; regenerate manifests instead of editing the generated copies.

Deploy

All four vLLM targets read the checkpoint from the model-cache PVC with HF_HUB_OFFLINE=1. Create and populate the PVC with the recipe’s storage manifest and model-download job:

kubectl apply -f recipes/glm-5.3-flash/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/glm-5.3-flash/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s

Apply the manifest or Kustomize overlay for your target. Run these commands from the repository root.

kubectl apply -f recipes/glm-5.3-flash/vllm/agg-h200-agentic/deploy.yaml -n ${NAMESPACE}
kubectl apply -f recipes/glm-5.3-flash/vllm/disagg-h200-agentic/deploy.yaml -n ${NAMESPACE}
kubectl apply -f recipes/glm-5.3-flash/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
kubectl apply -f recipes/glm-5.3-flash/vllm/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE}

Reuse shared-model-cache if it exists. For a new cache, set the storage class in the SGLang storage manifest before applying it. The download Job downloads the pinned NVFP4 revision, which differs from the vLLM checkpoint:

kubectl apply -f recipes/glm-5.3-flash/sglang/model-cache/model-cache.yaml -n "$NAMESPACE"
kubectl apply -f recipes/glm-5.3-flash/sglang/model-cache/model-download.yaml -n "$NAMESPACE"
kubectl wait --for=condition=Complete job/glm53-flash-nvfp4-model-download \
  -n "$NAMESPACE" --timeout=7200s

All SGLang workers resolve --model-path RadixArk/GLM-5.3-Flash-NVFP4 and the pinned --revision from the shared cache with offline mode enabled. All SGLang frontends obtain model metadata from workers without mounting the cache.

export RECIPE=recipes/glm-5.3-flash/sglang/agg-gb200-agentic
export RECIPE=recipes/glm-5.3-flash/sglang/disagg-gb200-agentic

The base deployment runs two prefill workers and one decode worker (2P1D), using 12 GB200 GPUs in one NVLink clique.

Render and apply the selected GB200 overlay:

kustomize build "$RECIPE/kustomize/overlays/gb200" > /tmp/glm53-sglang.yaml
kubectl apply --dry-run=server -f /tmp/glm53-sglang.yaml -n "$NAMESPACE"
kubectl apply -f /tmp/glm53-sglang.yaml -n "$NAMESPACE"
kubectl get dynamographdeployment,pods -n "$NAMESPACE"

The overlay creates a ComputeDomain and connects its channel to the worker claims. Alternatively, apply the generated "$RECIPE/deploy-gb200.yaml" manifest. The SGLang README documents the base/component layout and regeneration commands.

Render and apply the B200 nscale overlay:

export RECIPE=recipes/glm-5.3-flash/sglang/agg-b200-agentic
kustomize build "$RECIPE/kustomize/overlays/nscale" > /tmp/glm53-sglang-b200.yaml
kubectl apply --dry-run=server -f /tmp/glm53-sglang-b200.yaml -n "$NAMESPACE"
kubectl apply -f /tmp/glm53-sglang-b200.yaml -n "$NAMESPACE"
kubectl get dynamographdeployment,pods -n "$NAMESPACE"

Alternatively, apply the generated "$RECIPE/deploy-nscale.yaml" manifest. See the SGLang README for overlay settings and regeneration commands.

First worker launch loads the model and captures CUDA graphs and can take tens of minutes.

Smoke Test

Send a test request to verify the deployment serves traffic. First forward the frontend port for your target:

kubectl port-forward svc/glm53-flash-agg-frontend 8000:8000 -n ${NAMESPACE}
kubectl port-forward svc/glm53-flash-disagg-frontend 8000:8000 -n ${NAMESPACE}
kubectl port-forward svc/glm53-flash-agg-gb200-agentic-frontend 8000:8000 -n ${NAMESPACE}
kubectl port-forward svc/glm53-flash-disagg-gb200-frontend 8000:8000 -n ${NAMESPACE}

All vLLM targets serve under the same model name, zai-org/GLM-5.3-Flash:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"zai-org/GLM-5.3-Flash","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":64}'

GLM-5.3-Flash reasons before answering. The deployments support streaming chat, completions, tool calls, one image per prompt, and schema-constrained output through the glm45 reasoning and glm47 tool-call parsers.

Wait for the frontend and all workers to become ready, then forward the frontend Service for the selected topology:

kubectl port-forward svc/glm53-flash-sglang-agg-gb200-frontend 8000:8000 -n "$NAMESPACE"
kubectl port-forward svc/glm53-flash-sglang-disagg-gb200-frontend 8000:8000 -n "$NAMESPACE"

Wait for both Pods to become ready, then forward the B200 frontend:

kubectl port-forward svc/glm53-flash-sglang-agg-b200-frontend 8000:8000 -n "$NAMESPACE"

In another terminal, verify registration and send a request using the SGLang served model name:

curl --fail-with-body http://localhost:8000/v1/models
curl --fail-with-body http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"RadixArk/GLM-5.3-Flash-NVFP4","messages":[{"role":"user","content":"What is 2 + 2?"}],"max_tokens":128,"temperature":0}'

The SGLang workers use the glm45 reasoning parser and glm47 tool-call parser. All three recipes enable --enable-multimodal and --mm-enable-dp-encoder. Aggregated workers enable --frontend-decoding, so the Dynamo frontend decodes images and videos before passing them to the backend. Disaggregated workers omit --frontend-decoding on both prefill and decode workers and support text and image input only.

Video input is unavailable in the SGLang disaggregated recipe because frontend decoding is unsupported on this path and the runtime image omits SGLang’s video decoder dependencies.

Benchmark

Use the AIPerf benchmark guide and benchmark manifest for agentic text trace replay.

Synthetic Acceptance Results

These measurements describe the source configurations with simulated acceptance length 2.5657, method match-expected, and token mode real-draft-token. The default deployment overlays use real speculative verification. To configure synthetic acceptance, follow the compose, dry-run, and apply procedure with the optional component for B200 aggregated, GB200 aggregated, or GB200 disaggregated. Wait for replacement workers to become ready before measurement.

Set the CONCURRENCY environment variable in the benchmark Job to 56 for either aggregated target or 128 for GB200 disaggregated, as listed below. The Job’s default is 64 and does not match these rows. Select the matching endpoint and frontend pod affinity using the benchmark guide.

WorkloadFrameworkRecipeSKUCONCURRENCYSystem output tok/s/GPUUser output tok/s (P50)TTFT P50 (ms)
Agentic (15% subset)SGLangAggregated (1 worker)B20056579.150.7189
Agentic (15% subset)SGLangAggregated (1 worker)GB20056590.3451.05316
Agentic (15% subset)SGLangDisaggregated (2P1D)GB200128619.780.14,680

Compare All Targets

vLLM

All four vLLM targets serve the same FP8-quantized GLM-5.3-Flash checkpoint on vLLM with BF16 KV cache. The GB200 aggregated target and both H200 targets enable prefix caching. The H200 targets add MTP7 speculative decoding, DeepGEMM, and FLASHMLA_SPARSE attention. The GB200 targets use MNNVL for TP4 collectives.

GB200 AggregatedGB200 DisaggregatedH200 AggregatedH200 Disaggregated
TopologyDynamoGraphDeploymentDynamoGraphDeployment, 1P1DDynamoGraphDeploymentDynamoGraphDeployment, 1P1D
GPUs4x GB2004x GB200 prefill + 4x GB200 decode8x H2008x H200 prefill + 8x H200 decode
ParallelismTP4TP4 prefill / TP4 decodeTP8TP8 prefill / TP8 decode
PrecisionFP8 checkpoint + BF16 KVFP8 checkpoint + BF16 KVFP8 checkpoint + BF16 KVFP8 checkpoint + BF16 KV
SpeculationNoneNoneMTP7MTP7 on prefill and decode
Maximum context1,048,5761,048,5761,048,5761,048,576
Maximum sequences25632 prefill / 256 decode3216 prefill / 128 decode
Maximum batched tokens8,19216,384 prefill / 8,192 decode8,1928,192 prefill / 2,048 decode
GPU memory utilization0.920.92 prefill / 0.92 decode0.800.80 prefill / 0.85 decode
KV transferN/ANIXL/UCX over TCPN/ANIXL with GPU buffers and RDMA resources
RoutingKV-awareKV-awareRound-robinKV-aware

SGLang

All three SGLang targets use the pinned NVFP4 checkpoint and FP8 E4M3 KV cache. Worker Pods request 16 CPUs and four GPUs, with a 64 GiB shared-memory volume. Aggregated workers and disaggregated prefill workers have a host-memory request and limit of 448 GiB.

SettingB200 AggregatedGB200 AggregatedGB200 Disaggregated
Topology1 worker on one amd64 node1 worker on arm642 prefill + 1 decode on arm64
GPUs4412
ParallelismTP4TP4Prefill TP4/EP4, decode TP4
SpeculationEAGLE, 3 steps / 4 draft tokens, ReplaySSMSameSame, decode only
HiCache CPU capacity100 GB per TP rank; ~400 GB per worker PodSame100 GB per prefill TP rank; ~400 GB per prefill Pod; none on decode
Host memory request / limit per worker Pod448 GiB / 448 GiB448 GiB / 448 GiBPrefill: 448 GiB / 448 GiB; decode: 256 GiB request, no explicit limit
Maximum running requests646464 per prefill worker / 128 decode
Maximum decode CUDA graph batch size6464128 on decode
Chunked prefill size16,384 tokens16,384 tokens16,384 tokens
Static memory fraction0.700.70Prefill 0.70 / decode 0.82
RoutingRound-robinRound-robinKV-aware with worker KV events
KV transferNot applicableNot applicableMooncake over NVLink with MNNVL
ComputeDomainNoneCreated by overlayCreated by overlay
Overlaynscalegb200gb200
Input modalitiesText, images, videoText, images, videoText, images

No controlled feature benchmark is linked to this recipe. Browse the Feature Benchmarks for comparable KV-routing, disaggregation, and speculative-decoding studies.

Notes

vLLM

  • The H200 targets enable prefix caching with a match unit of 128 and use MTP with seven speculative tokens.
  • The H200 disaggregated target sets the same MTP configuration, VLLM_SSM_CONV_STATE_LAYOUT=DS, and VLLM_KV_CACHE_LAYOUT=HND on prefill and decode. These values must match for the NIXL compatibility handshake and KV layout.
  • The GB200 targets use MNNVL for TP4 collectives. The disaggregated target excludes cuda_ipc from UCX and transfers KV data over cuda_copy and TCP because the GLM image’s UCX build lacks MNNVL IPC support.
  • To enable MNNVL IPC for GB200 KV transfer, rebuild on the vLLM runtime base and set UCX_TLS=cuda_copy,cuda_ipc,tcp with UCX_CUDA_IPC_ENABLE_MNNVL=y.
  • A sitecustomize.py overlay disables cuDNN to avoid a convolution crash during multimodal initialization with the GLM-specific vLLM image.
  • n>1 requests are not supported by the disaggregated targets.

SGLang

  • The pinned SGLang runtime image includes the cuDNN workaround for the vision encoder downsampling operation.
  • All worker bases set DYN_HEALTH_CHECK_REQUEST_TIMEOUT=300 and an explicit /live liveness probe on port system, with period 5 seconds, timeout 4 seconds, and failure threshold 60. GB200 components provide placement and ComputeDomain wiring; startup and readiness probes use operator defaults.
  • GB200 disaggregated routing uses worker KV events. Both prefill and decode workers set --kv-events-config to a ZMQ publisher on tcp://*:5557 with topic kv-events. The router block size and SGLang page size are both 64.
  • All three SGLang default overlays omit the source configurations’ SGLANG_SIMULATE_ACC_* variables. Add the optional component only when reproducing synthetic acceptance measurements.

Source