跳到导航
Model RecipesSK Telecom

A.X-K2

Serve A.X-K2 with Dynamo and vLLM on B200 or H200, aggregated or disaggregated.

以 Markdown 格式查看

Deploy SK Telecom’s A.X-K2 with NVIDIA Dynamo and vLLM. B200 targets use NVFP4 weights; H200 targets use FP8 weights. H200 targets use BF16 KV cache; B200 targets use FP8 KV cache. All use EAGLE3 with three speculative tokens.

Choose your deployment target

GPU
Topology
Checkpoint skt/A.X-K2-NVFP4Hardware 8x B200, two TP4 aggregate workersPrecision NVFP4 weights, FP8 KV cacheSpeculation EAGLE3, 3 tokensFlashInfer autotuning Disabled
Checkpoint skt/A.X-K2-NVFP4Hardware 12x B200, two TP4 prefill workers and one TP4 decode workerPrecision NVFP4 weights, FP8 KV cacheSpeculation EAGLE3, 3 tokens on both rolesFlashInfer autotuning Disabled on prefill and decode
Checkpoint skt/A.X-K2Hardware 32x H200, four TP8 aggregate workers, expert parallelism enabledPrecision FP8 weights, BF16 KV cacheSpeculation EAGLE3, 3 tokens, draft KV dtype auto
Checkpoint skt/A.X-K2Hardware 32x H200, two TP8/DP1 prefill workers and two TP8 decode workers, expert parallelism enabledPrecision FP8 weights, BF16 KV cacheSpeculation EAGLE3, 3 tokens, draft KV dtype auto

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed.
  • A ReadWriteMany storage class for model weights.
  • A Hugging Face token with access to skt/A.X-K2, skt/A.X-K2-NVFP4, and skt/A.X-K2-EAGLE3.

Each worker needs four B200 GPUs and 400 GiB of host memory: eight GPUs total for aggregated serving or twelve for disaggregated serving.

Each target needs four workers with eight H200 GPUs each (32 GPUs total). Each worker requests 512 GiB of host memory with a 768 GiB limit. Use a cluster overlay to bind H200 scheduling and expose your RDMA devices and network interfaces; the generic manifests do not supply site bindings.

For H200 results, use the benchmark configuration described below.

Disaggregated serving also requires InfiniBand with one rdma/shared_ib device resource available per worker for NIXL/UCX KV transfer.

Deploy

Run commands from the repository root. Set CONTEXT and NAMESPACE to your cluster context and namespace, and set HF_TOKEN to your Hugging Face token. Create the namespace and token secret:

kubectl --context "${CONTEXT}" create namespace "${NAMESPACE}"
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="${HF_TOKEN}"

Set storageClassName in model-cache.yaml to your cluster’s ReadWriteMany storage class. Create the PVC and download the three pinned checkpoints:

kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply -f recipes/ax-k2/model-cache/model-cache.yaml
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply -f recipes/ax-k2/model-cache/model-download.yaml
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" wait --for=condition=Complete \
  job/axk2-model-download --timeout=14400s

All profiles and the download job mount model-cache. To use an existing populated PVC, set claimName in the job and selected deployment source to that PVC and skip creating a new claim. Regenerate the selected manifest after editing its Kustomize source.

Deploy two aggregate workers:

export DGD=axk2-agg-b200-chat
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply \
  -f recipes/ax-k2/vllm/agg-b200-chat/deploy-generic.yaml

The manifest is generated from Kustomize sources. To change a setting, edit kustomize/base/deploy.yaml and regenerate:

python3 scripts/kustomize-matrix.py unfold recipes/ax-k2/vllm/agg-b200-chat/.kustomize-matrix.yaml
python3 scripts/kustomize-matrix.py render recipes/ax-k2/vllm/agg-b200-chat/.kustomize-matrix.yaml

Deploy two prefill workers and one decode worker:

export DGD=axk2-disagg-b200-chat
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply \
  -f recipes/ax-k2/vllm/disagg-b200-chat/deploy-generic.yaml

The manifest is generated from Kustomize sources. To change a setting, edit kustomize/base/deploy.yaml and regenerate:

python3 scripts/kustomize-matrix.py unfold recipes/ax-k2/vllm/disagg-b200-chat/.kustomize-matrix.yaml
python3 scripts/kustomize-matrix.py render recipes/ax-k2/vllm/disagg-b200-chat/.kustomize-matrix.yaml

Deploy four aggregate workers after applying your cluster bindings to the Kustomize sources and regenerating the manifest:

python3 scripts/kustomize-matrix.py unfold recipes/ax-k2/vllm/agg-h200/.kustomize-matrix.yaml
python3 scripts/kustomize-matrix.py render recipes/ax-k2/vllm/agg-h200/.kustomize-matrix.yaml
export DGD=axk2-vllm-agg-h200
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply \
  -f recipes/ax-k2/vllm/agg-h200/deploy-generic.yaml

Deploy two prefill workers and two decode workers after applying your cluster bindings to the Kustomize sources and regenerating the manifest:

python3 scripts/kustomize-matrix.py unfold recipes/ax-k2/vllm/disagg-h200/.kustomize-matrix.yaml
python3 scripts/kustomize-matrix.py render recipes/ax-k2/vllm/disagg-h200/.kustomize-matrix.yaml
export DGD=axk2-vllm-disagg-h200
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply \
  -f recipes/ax-k2/vllm/disagg-h200/deploy-generic.yaml

Smoke Test

Wait for the selected deployment, then forward its frontend port:

kubectl --context "${CONTEXT}" -n "${NAMESPACE}" wait --for=condition=Ready \
  "dynamographdeployment/${DGD}" --timeout=7200s
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" port-forward \
  "service/${DGD}-frontend" 8000:8000

In another terminal, verify model discovery and a completion:

curl --fail http://localhost:8000/v1/models
curl --fail http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"skt/A.X-K2-NVFP4","messages":[{"role":"user","content":"What is 2 + 2?"}],"temperature":0,"max_tokens":4096,"stream":false}'

Model discovery should list skt/A.X-K2-NVFP4; the completion should contain a choices array. Reasoning uses the deepseek_v3 parser, and tool calling uses the hermes parser.

In another terminal, verify model discovery and a completion:

curl --fail http://localhost:8000/v1/models
curl --fail http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"A.X-K2","messages":[{"role":"user","content":"What is 2 + 2?"}],"temperature":0,"max_tokens":4096,"stream":false}'

Model discovery should list A.X-K2; the completion should contain a choices array. Reasoning uses the deepseek_v3 parser, and tool calling uses the hermes parser.

Benchmark

The AIPerf workflow replays an 8K-input / 1K-output chat trace with 70% KV reuse against the aggregated profile at concurrency 32. It preserves expanded configurations, raw reports, and frontend metrics. Keep the default real EAGLE3 acceptance configuration for accuracy evaluation.

Before starting the benchmark, follow the workflow’s trace-staging instructions to copy the Git LFS chat trace from your checkout onto the model-cache PVC. The job reads it through TRACE_FILE and verifies its SHA-256 and request counts before sending traffic.

The checked-in benchmark job targets the aggregate frontend. To benchmark the disaggregated profile, update its endpoint and frontend selector to this deployment before applying the job.

Performance Results

Benchmarking uses synthetic EAGLE3 acceptance length 2.12.

WorkloadFrameworkRecipeSKUConcurrencySystem output tok/s/GPUUser output tok/s (mean)TTFT P50 (seconds)
Chat (15% subset)vLLMAggregated (2 workers)B2001676.4955.830.525
Chat (15% subset)vLLMDisaggregated (2P1D)B2001661.97833.879

Configure the H200 AIPerf Job for the running deployment using the benchmark instructions.

Run with 32 H200 GPUs, 1.4.1-a.x-k2-post.1, FP8 weights, BF16 KV cache, flashinfer_cutlass, FLASH_ATTN_MLA_SPARSE, and EAGLE3 k=3 with real acceptance. The checked-in AIPerf Job uses different sweep settings from the measurements below.

SettingValue
Topology4×TP8 aggregated or 2P2D TP8 (32 H200 GPUs)
WorkloadW1: 10,240 shared-prefix + 6,144 unique input; 1,024 output; 32 prefixes
Cache / RoutingKV-aware; fresh workers, cache retained within each sweep
Engine Limits262,144 context; batch 8,192; 64 sequences
KV Capacity64-token blocks; 465,856 tokens per worker
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" apply -f recipes/ax-k2/perf/h200/perf.yaml
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" wait \
  --for=condition=Complete job/axk2-h200-perf --timeout=21600s
kubectl --context "${CONTEXT}" -n "${NAMESPACE}" logs job/axk2-h200-perf -c aiperf

H200 Performance Results

RecipeConcurrencySystem output tok/sSystem output tok/s/GPUUser output tok/s (mean)TPOT p50 (ms)TPOT p99 (ms)TTFT p50 (ms)
Aggregated (4 workers)16896.0928.0060.8416.8720.87655.40
Disaggregated (2P2D)16883.4027.6160.7317.0920.73817.80
Aggregated (4 workers)321,388.6343.3946.7222.2727.16652.25
Disaggregated (2P2D)321,483.0846.3551.1420.2723.58902.83
Aggregated (4 workers)641,941.5260.6733.2131.6442.08678.50
Disaggregated (2P2D)642,141.4966.9238.8626.5536.111,629.23
Aggregated (4 workers)1282,646.1382.6922.5946.9961.10707.32
Disaggregated (2P2D)1282,034.0263.5630.7334.2247.2128,194.28

Recommendation

  • Low concurrency or TTFT-first: prefer aggregated serving; 2P2D has no throughput gain at c16.
  • Throughput/TPOT-first: consider 2P2D at c64, with TTFT p50 of 1.63 seconds.
  • Higher concurrency: prefer aggregated at c128; 2P2D has lower throughput and TTFT p50 of 28.19 seconds.

Compare All Targets

B200

SettingAggregatedDisaggregated
Workers2 aggregate2 prefill + 1 decode
Total GPUs8x B20012x B200
ParallelismTP4, DP1, expert parallelism disabledSame on both roles
PrecisionNVFP4 weights, FP8 KV cacheSame on both roles
AttentionFLASHINFER_MLA_SPARSESame on both roles
FlashInfer autotuningDisabledDisabled on prefill and decode
EAGLE33 speculative tokens3 speculative tokens on both roles
Async schedulingEnabledPrefill disabled, decode enabled
Context length262,144 tokens262,144 tokens
RoutingKV-awareKV-aware
KV transferN/ANIXL/UCX over InfiniBand

H200

SettingAggregatedDisaggregated
Workers4 aggregate2 prefill + 2 decode
Total GPUs32x H20032x H200
ParallelismTP8 + expert parallelismPrefill TP8/DP1 + expert parallelism; decode TP8 + expert parallelism
Checkpointskt/A.X-K2, FP8 weightsSame
KV Cachebfloat16Same on both roles
EAGLE33 speculative tokens, draft KV dtype autoSame on both roles
MoE Backendflashinfer_cutlassSame on both roles
Attention BackendFLASH_ATTN_MLA_SPARSESame on both roles
Max Sequences per Engine6464 on both roles
Context Length262,144 tokensSame
RoutingKV-awareKV-aware
KV TransferN/ANIXL/UCX over RDMA

If model initialization runs out of memory, lower --max-model-len to 32768 on every worker.

Source