> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # K-EXAONE 2.0 This recipe serves `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4` — a 764.5B-parameter MoE with \~37B active per token, 256 experts at top-8 routing, and hybrid attention (20 full-attention plus 58 sliding-attention layers) — on B200 with vLLM. Multi-Token Prediction is enabled, and the MoE kernel is pinned to FLASHINFER\_CUTLASS for correctness. Pick aggregated or disaggregated; every command on this page updates to match. Choose your deployment target Topology Aggregated Recommended Disaggregated (1P1D) **Checkpoint** LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4 **Precision** NVFP4 W4A4 + FP8 KV cache **GPUs** 4x B200 **Parallelism** TP4 **Spec decoding** MTP, draft length 2 **Workload** 8K ISL / 1K OSL chat trace, designed for \~70% prefix reuse (\~8.8% achieved here) **Checkpoint** LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4 **Precision** NVFP4 W4A4 + FP8 KV cache **GPUs** 8x B200, 4 prefill + 4 decode **Parallelism** TP4 prefill, TP4 decode **KV transfer** NIXL over UCX, InfiniBand RDMA **Workload** 8K ISL / 1K OSL chat trace, \~70% prefix reuse ## Prerequisites * A Kubernetes cluster with the Dynamo Operator. * A `ReadWriteMany` storage class with at least 700 GB free — the checkpoint is \~530 GB on disk. * A Hugging Face token with access to `LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4`. - **4x B200** on a single node. * **8x B200**, ideally on a single node. Prefill and decode are co-located by a *preferred* pod affinity so the KV hop stays on one fabric where capacity allows; a required affinity deadlocks the operator's gang scheduling. KV moves over IB RDMA, so a split placement still works, just slower. * An **RDMA device plugin** exposing InfiniBand HCAs as a Kubernetes extended resource. `deploy-ib.yaml` already carries the fabric settings — do not add your own on top. Review them first; they are cluster-specific. See [Notes](#notes). ## Deploy Create the namespace, token secret, and storage: ```bash export NAMESPACE=your-namespace kubectl create namespace ${NAMESPACE} kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=${HF_TOKEN} -n ${NAMESPACE} # edit storageClassName to match your cluster first kubectl apply -f recipes/k-exaone-2.0/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/k-exaone-2.0/model-cache/model-download.yaml -n ${NAMESPACE} kubectl wait --for=condition=complete job/model-download -n ${NAMESPACE} --timeout=6h ``` ```bash kubectl apply -f recipes/k-exaone-2.0/vllm/agg-b200-chat/deploy-generic.yaml -n ${NAMESPACE} kubectl wait --for=condition=Ready dgd/k-exaone-2-vllm-b200-agg-chat -n ${NAMESPACE} --timeout=7200s ``` ```bash kubectl apply -f recipes/k-exaone-2.0/vllm/disagg-b200-chat/deploy-ib.yaml -n ${NAMESPACE} kubectl wait --for=condition=Ready dgd/k-exaone-2-vllm-b200-disagg-chat -n ${NAMESPACE} --timeout=7200s ``` First start takes **40–120 minutes**: 53 shards load, then autotune, then CUDA-graph capture. Silence is not a hang — watch the worker log for shard progress. ## Smoke Test ```bash kubectl port-forward svc/k-exaone-2-vllm-b200-agg-chat-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/k-exaone-2-vllm-b200-disagg-chat-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4", "messages": [{"role":"user","content":"In 2-3 sentences, explain why the daytime sky is blue."}], "max_tokens": 2048, "temperature": 0.6, "top_p": 0.95 }' | jq -r '.choices[0].message.content // .choices[0].message.reasoning_content' ``` This is a reasoning model. Use temperature 0.6 rather than greedy decoding, give it a generous token budget, and read `.content // .reasoning_content` — reasoning consumes the budget before the final answer, so a short cap prints `null` on a perfectly healthy deployment. ## Benchmark ```bash kubectl apply -f recipes/k-exaone-2.0/perf/perf.yaml -n ${NAMESPACE} ``` Trace replay with AIPerf against the Mooncake 8K/1K/70%-reuse chat trace. See [`perf/README.md`](https://github.com/ai-dynamo/dynamo/blob/main/recipes/k-exaone-2.0/perf/README.md) for staging the trace, running a concurrency sweep, and fetching artifacts. ## Expected Performance Mooncake chat trace replay, 1,805 requests, at the highest concurrency meeting both E2E ≥ 50 tok/s/user and TTFT p50 \< 5 s. | Workload | Recipe | Framework | SKU | Concurrency | System output tok/s/GPU | User output tok/s (P50) | TTFT P50 (ms) | | ---------------- | --------------- | --------- | ---- | ----------- | ----------------------- | ----------------------- | ------------- | | Chat 8K/1K trace | `agg-b200-chat` | vLLM | B200 | 7 | 87 | 51.4 | 291 | ITL p50 is 19.15 ms. Measured KV reuse on this workload is 8.8%: the trace's working set far exceeds TP4's KV capacity, so blocks are evicted before they can be reused. Mooncake chat trace replay, 1,805 requests, at the highest concurrency meeting the same gate. | Workload | Recipe | Framework | SKU | Concurrency | System output tok/s/GPU | User output tok/s (P50) | TTFT P50 (ms) | | ---------------- | ------------------ | --------- | ---- | ----------- | ----------------------- | ----------------------- | ------------- | | Chat 8K/1K trace | `disagg-b200-chat` | vLLM | B200 | 14 | 85 | 54.9 | 2,230 | ITL p50 is 15.98 ms. Throughput per GPU is level with the aggregated target, for twice the GPUs. Choose this target for lower per-token latency and independent prefill/decode scaling, not for throughput. 38 of the trace's 1,805 requests exceed the 262,144-token context ceiling and return HTTP 400; they are excluded from the metrics. ## Compare All Targets | Target | GPUs | Topology | Parallelism | Spec decoding | KV transfer | | ------------- | ------- | ---------- | ----------- | ----------------------- | --------------------- | | Aggregated | 4x B200 | Aggregated | TP4 | MTP, DL=2 | n/a | | Disaggregated | 8x B200 | 1P1D | TP4 / TP4 | MTP, DL=2 on both roles | NIXL/UCX over IB RDMA | ## Supported Features | Feature | | Notes | | -------------------------- | - | -------------------------------------------------------------------------------- | | NVFP4 (W4A4) weights | ✅ | the FLASHINFER\_CUTLASS MoE backend is **mandatory** — see [Notes](#notes) | | FP8 KV cache | ✅ | the checkpoint declares `kv_cache_quant_algo: FP8`; it is the default path | | Prefix caching | ✅ | on by default | | Speculative decoding (MTP) | ✅ | `--spec-method mtp --spec-tokens 2` | | Reasoning parser | ✅ | `--dyn-reasoning-parser qwen3` | | Tool calling | ✅ | `--dyn-tool-call-parser qwen3_coder` | | Disaggregated serving | ✅ | NIXL over UCX; needs an RDMA device plugin | | KV-aware routing | ✅ | wired end to end; inert at the shipped `replicas: 1`, correct when you scale out | | 262,144-token context | ✅ | the checkpoint's native window, served in full with no rope scaling | | Expert parallelism | ➖ | no measurable gain at TP4; not enabled | ## Notes * **The MoE backend pin is a correctness requirement.** vLLM's `auto` selection picks FLASHINFER\_TRTLLM, which silently corrupts long-form output on this checkpoint — fluent, plausible, wrong, with no error. Both manifests pin `--kernel-config '{"moe_backend":"FLASHINFER_CUTLASS"}'`; the pinned kernel is slower and that cost is inside the figures above. * **Use the `--dyn-*` parser flags.** Plain `--reasoning-parser` and `--tool-call-parser` configure the engine only and never reach the Dynamo frontend, so tool calling silently does nothing. * **TP4 is both the floor and the optimum.** \~530 GB of weights does not fit TP2 on 180 GB B200s, and TP8 is slower per GPU: this model is communication-bound rather than KV-capacity-bound. * **Disaggregated: prefill and decode must agree** on speculative decoding and block size. A mismatch changes KV block geometry and produces silent garbage rather than an error. `--max-num-seqs` is the deliberate exception: 32 on prefill, 256 on decode. * **`deploy-ib.yaml` already carries the InfiniBand settings, and they are cluster-specific.** Check both against your cluster before deploying, and do not add your own fabric configuration on top. The RDMA extended resource is requested as `rdma/shared_ib`; other clusters expose `rdma/ib` or `rdma/rdma_shared_device_a`, and a shared flavour is preferable — with an exclusive-mode resource, two co-located workers each claiming HCAs can deadlock NCCL bootstrap. `UCX_TLS` is set to `rc_x,rc,cuda_copy,cuda_ipc`, which are the InfiniBand verbs transports and are wrong on a RoCE or EFA fabric; `tcp` is omitted deliberately so a misconfigured fabric fails loudly instead of silently falling back to a slow transport. Verify the transport engaged before trusting any measurement: ```bash kubectl exec -n ${NAMESPACE} -- \ curl -s localhost:9090/metrics | grep vllm:nixl_xfer_time_seconds ``` Divide `_sum` by `_count`: a \~1 GB KV transfer should take milliseconds, not seconds. * **KV-aware routing is wired end to end but inert at `replicas: 1`.** The frontend ships `--router-mode kv --router-kv-events` and the worker (prefill, in the disaggregated recipe) ships `--kv-events-config`, so the block index is published and consumed. One worker is one destination, so raising `replicas` is all that is needed — no flag edit. On this workload achieved reuse is capacity-capped, so expect correct scale-out behaviour rather than a throughput gain. * **There is no cache-clearing endpoint.** `dynamo.vllm` runs the Dynamo worker rather than vLLM's OpenAI server, so `POST /reset_prefix_cache` is not served anywhere and `VLLM_SERVER_DEV_MODE` has no effect. `perf.yaml` runs one `CONCURRENCY` value and does not salt its requests, so restart the workers between points to clear engine KV — otherwise each point inherits the previous point's cache. * **Context length is 262,144**, the model's native maximum. The checkpoint uses unscaled rope, so longer contexts would require a rope-scaling override with unvalidated accuracy. ## Source * [Recipe directory](https://github.com/ai-dynamo/dynamo/tree/main/recipes/k-exaone-2.0) * [Aggregated manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/k-exaone-2.0/vllm/agg-b200-chat/deploy-generic.yaml) * [Disaggregated manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/k-exaone-2.0/vllm/disagg-b200-chat/deploy-ib.yaml) * [Model card](https://huggingface.co/LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4)