K-EXAONE 2.0

This recipe serves LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4 — a 764.5B-parameter MoE with ~37B active per token, 256 experts at top-8 routing, and hybrid attention (20 full-attention plus 58 sliding-attention layers) — on B200 with vLLM. Multi-Token Prediction is enabled, and the MoE kernel is pinned to FLASHINFER_CUTLASS for correctness. Pick aggregated or disaggregated; every command on this page updates to match.

Prerequisites

• A Kubernetes cluster with the Dynamo Operator.
• A ReadWriteMany storage class with at least 700 GB free — the checkpoint is ~530 GB on disk.
• A Hugging Face token with access to LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4.

Deploy

Create the namespace, token secret, and storage:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=${HF_TOKEN} -n ${NAMESPACE}
# edit storageClassName to match your cluster first
kubectl apply -f recipes/k-exaone-2.0/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/k-exaone-2.0/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=complete job/model-download -n ${NAMESPACE} --timeout=6h

kubectl apply -f recipes/k-exaone-2.0/vllm/agg-b200-chat/deploy-generic.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Ready dgd/k-exaone-2-vllm-b200-agg-chat -n ${NAMESPACE} --timeout=7200s

kubectl apply -f recipes/k-exaone-2.0/vllm/disagg-b200-chat/deploy-ib.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Ready dgd/k-exaone-2-vllm-b200-disagg-chat -n ${NAMESPACE} --timeout=7200s

First start takes 40–120 minutes: 53 shards load, then autotune, then CUDA-graph capture. Silence is not a hang — watch the worker log for shard progress.

Smoke Test

kubectl port-forward svc/k-exaone-2-vllm-b200-agg-chat-frontend 8000:8000 -n ${NAMESPACE}

kubectl port-forward svc/k-exaone-2-vllm-b200-disagg-chat-frontend 8000:8000 -n ${NAMESPACE}

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4",
  "messages": [{"role":"user","content":"In 2-3 sentences, explain why the daytime sky is blue."}],
  "max_tokens": 2048, "temperature": 0.6, "top_p": 0.95
}' | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content'

This is a reasoning model. Use temperature 0.6 rather than greedy decoding, give it a generous token budget, and read .content // .reasoning_content — reasoning consumes the budget before the final answer, so a short cap prints null on a perfectly healthy deployment.

Benchmark

kubectl apply -f recipes/k-exaone-2.0/perf/perf.yaml -n ${NAMESPACE}

Trace replay with AIPerf against the Mooncake 8K/1K/70%-reuse chat trace. See perf/README.md for staging the trace, running a concurrency sweep, and fetching artifacts.

Expected Performance

Compare All Targets

Supported Features

Notes

• The MoE backend pin is a correctness requirement. vLLM’s auto selection picks FLASHINFER_TRTLLM, which silently corrupts long-form output on this checkpoint — fluent, plausible, wrong, with no error. Both manifests pin --kernel-config '{"moe_backend":"FLASHINFER_CUTLASS"}'; the pinned kernel is slower and that cost is inside the figures above.

• Use the --dyn-* parser flags. Plain --reasoning-parser and --tool-call-parser configure the engine only and never reach the Dynamo frontend, so tool calling silently does nothing.

• TP4 is both the floor and the optimum. ~530 GB of weights does not fit TP2 on 180 GB B200s, and TP8 is slower per GPU: this model is communication-bound rather than KV-capacity-bound.

• Disaggregated: prefill and decode must agree on speculative decoding and block size. A mismatch changes KV block geometry and produces silent garbage rather than an error. --max-num-seqs is the deliberate exception: 32 on prefill, 256 on decode.

• deploy-ib.yaml already carries the InfiniBand settings, and they are cluster-specific. Check both against your cluster before deploying, and do not add your own fabric configuration on top. The RDMA extended resource is requested as rdma/shared_ib; other clusters expose rdma/ib or rdma/rdma_shared_device_a, and a shared flavour is preferable — with an exclusive-mode resource, two co-located workers each claiming HCAs can deadlock NCCL bootstrap. UCX_TLS is set to rc_x,rc,cuda_copy,cuda_ipc, which are the InfiniBand verbs transports and are wrong on a RoCE or EFA fabric; tcp is omitted deliberately so a misconfigured fabric fails loudly instead of silently falling back to a slow transport. Verify the transport engaged before trusting any measurement:

  kubectl exec <decode-worker> -n ${NAMESPACE} -- \
    curl -s localhost:9090/metrics | grep vllm:nixl_xfer_time_seconds

  Divide _sum by _count: a ~1 GB KV transfer should take milliseconds, not seconds.

• KV-aware routing is wired end to end but inert at replicas: 1. The frontend ships --router-mode kv --router-kv-events and the worker (prefill, in the disaggregated recipe) ships --kv-events-config, so the block index is published and consumed. One worker is one destination, so raising replicas is all that is needed — no flag edit. On this workload achieved reuse is capacity-capped, so expect correct scale-out behaviour rather than a throughput gain.

• There is no cache-clearing endpoint. dynamo.vllm runs the Dynamo worker rather than vLLM’s OpenAI server, so POST /reset_prefix_cache is not served anywhere and VLLM_SERVER_DEV_MODE has no effect. perf.yaml runs one CONCURRENCY value and does not salt its requests, so restart the workers between points to clear engine KV — otherwise each point inherits the previous point’s cache.

• Context length is 262,144, the model’s native maximum. The checkpoint uses unscaled rope, so longer contexts would require a rope-scaling override with unvalidated accuracy.

Source

• Recipe directory
• Aggregated manifest
• Disaggregated manifest
• Model card