跳到导航
Model RecipesMotif Technologies

Motif-3 NVFP4

Deploy aggregated or 3P1D KV-aware Motif-3 serving on NVIDIA B200 GPUs.

以 Markdown 格式查看

Experimental. Serve Motif-3 NVFP4 with NVIDIA Dynamo and vLLM on B200. Choose one aggregated worker or three prefill workers and one decode worker (3P1D) with KV-aware routing. Each worker uses two GPUs (TP2), expert parallelism, and multi-token prediction with two speculative tokens (MTP2).

Choose your deployment target

Topology
Hardware 2x NVIDIA B200Serving One TP2 worker, expert parallelism, MTP2Cache NVFP4 weights, FP8 KV cache, shared model cache
Hardware 8x NVIDIA B200Serving 3 TP2 prefill + 1 TP2 decode, expert parallelism, MTP2Routing KV-aware, NIXL/UCX transfer, FP8 KV cache

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed and a shared ReadWriteMany model-cache volume.
  • A Hugging Face token with access to the checkpoint, used by the model-download Job.
  • Two available B200 GPUs on one node.
  • Eight available B200 GPUs. Each TP2 worker occupies two GPUs on one node; configure the fabric for NIXL/UCX transfer between nodes.

Run the following commands from the repository root. Create a namespace and download secret if they do not already exist:

export NAMESPACE=your-namespace
kubectl create namespace "${NAMESPACE}"
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="$HF_TOKEN" \
  -n "${NAMESPACE}"

Deploy

Reuse a populated shared-model-cache PVC when available. Otherwise, set storageClassName in the model-cache manifest, then create the cache and download the checkpoint. For Nscale, use storageClassName: vast.

kubectl apply -f recipes/motif-3/model-cache/model-cache.yaml -n "${NAMESPACE}"
kubectl apply -f recipes/motif-3/model-cache/model-download.yaml -n "${NAMESPACE}"
kubectl wait --for=condition=Complete job/model-download -n "${NAMESPACE}" --timeout=7200s

Apply the aggregated manifest:

kubectl apply -f recipes/motif-3/vllm/agg-b200-chat/base/deploy.yaml -n "${NAMESPACE}"
kubectl get dgd motif3-agg-b200 -n "${NAMESPACE}" -w

For Nscale B200 selection and GPU tolerations, apply the aggregated Kustomization instead:

kubectl apply -k recipes/motif-3/vllm/agg-b200-chat/kustomize -n "${NAMESPACE}"

Rendering the aggregated Kustomization requires Git and network access.

Configure GPU node selection and fabric resources for your cluster, then apply the generic 3P1D manifest:

kubectl apply -f recipes/motif-3/vllm/disagg-b200-chat/deploy-generic.yaml -n "${NAMESPACE}"
kubectl get dgd motif3-disagg-b200 -n "${NAMESPACE}" -w

For Nscale, use the optional site component to add B200 node selection and RDMA settings:

python3 scripts/kustomize-matrix.py compose \
  recipes/motif-3/vllm/disagg-b200-chat/kustomize/overlays/generic \
  recipes/motif-3/vllm/disagg-b200-chat/kustomize/components/nscale \
  | kubectl apply -f - -n "${NAMESPACE}"

Wait for the frontend and all four workers to become ready.

Smoke Test

Forward the frontend port for the selected target:

kubectl port-forward svc/motif3-agg-b200-frontend 8000:8000 -n "${NAMESPACE}"
kubectl port-forward svc/motif3-disagg-b200-frontend 8000:8000 -n "${NAMESPACE}"

In another terminal, verify model discovery and a completion:

curl --fail http://localhost:8000/v1/models
curl --fail http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Motif-Technologies/Motif-3-NVFP4",
    "messages": [{"role": "user", "content": "Write a one-sentence readiness check."}],
    "max_tokens": 64,
    "temperature": 0
  }'

Benchmark

Stage the complete 8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl trace at /shared-model-cache/traces/8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl.

The aggregated AIPerf Job uses concurrency 11:

kubectl apply -f recipes/motif-3/perf/perf.yaml -n "${NAMESPACE}"
kubectl logs -f job/motif3-aiperf-job -n "${NAMESPACE}"

Follow the disaggregated benchmark workflow to prepare the client and reset worker and router caches before each independent run.

The runner sends 32 warmup requests at concurrency 32, then measures all 1,805 trace rows at concurrency 64. It retains worker and router caches between phases and saves separate artifacts; warmup requests are excluded from the results.

Both targets default to real MTP verification. To use synthetic acceptance length 2.13 for benchmarking, change the SPECULATIVE_CONFIG ConfigMap key from speculative-config to speculative-config-synthetic in the deployment source, regenerate any derived manifest, and redeploy. For disaggregation, change both worker roles; see the recipe README for regeneration commands.

Expected Performance

These chat-trace results use synthetic MTP2 acceptance length 2.13. They do not measure real MTP acceptance or output quality. The default manifests use real MTP verification.

TTFT is Time To First Token. Per-user output rate excludes TTFT; throughput per GPU includes every serving GPU.

DeploymentGPUsConcurrencyOutput tokens/sec/GPUP50 tokens/sec/userP50 TTFT (ms)
Aggregated211293.9666.27388.18

Thirty-five requests exceeded the 262,144-token context limit.

DeploymentGPUsConcurrencyOutput tokens/sec/GPUP50 tokens/sec/userP50 TTFT (ms)
Disaggregated 3P1D, KV-aware864319.2651.494305.08

Compare All Targets

TargetWorkersB200 GPUsDeployment
Aggregated1 × TP22Generic · Nscale
Disaggregated, KV-aware3 prefill × TP2 + 1 decode × TP28Generic

Notes

  • Both targets use NVFP4 weights, FP8 KV cache, prefix caching, and a 262,144-token context limit. Frontends explicitly set --kv-cache-block-size 128, matching --block-size=128 on every worker.
  • Both targets use the public nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.0-motif-3-dev.2 image; no NGC API key or image-pull secret is required. The Hugging Face secret is used only to download the model.
  • Populate the shared model cache before starting the workers, which run in offline mode.
  • The disaggregated recipe has been validated with TP2 on both prefill and decode workers.

Source