> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Solar Open2 250B ## Profiles | Profile | Serving mode | GPUs | Context | | ------------------ | ------------------------------------------ | ---: | ------- | | `agg-b200-chat` | Aggregated, two replicas, KV-aware routing | 4 | 1M | | `disagg-b200-chat` | Disaggregated 1P:2D, round-robin routing | 8 | 1M | ## Prerequisites * Kubernetes cluster with B200 nodes and the Dynamo operator installed * A `ReadWriteMany` storage class. `model-cache.yaml` requests 1200Gi, which holds the checkpoint alongside benchmark traces and artifacts; the checkpoint itself is about 153 GB. * A Hugging Face token, as a secret named `hf-token-secret`, used by the download Job to fetch the checkpoint: ```bash kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN= ``` * For the disaggregated profile, an RDMA/InfiniBand device plugin exposing `rdma/ib` on the GPU nodes. If your plugin advertises a different resource name, substitute it in the worker resource requests. ## Prepare the model cache Edit `storageClassName` in `model-cache.yaml` first. ```bash kubectl apply -f recipes/solar-open2-250b/model-cache/model-cache.yaml kubectl apply -f recipes/solar-open2-250b/model-cache/model-download.yaml kubectl wait --for=condition=complete job/model-download --timeout=2h ``` The checkpoint is approximately 153 GB. Both profiles share this cache. ## Deploy Aggregated: ```bash kubectl apply -f recipes/solar-open2-250b/vllm/agg-b200-chat/deploy.yaml kubectl wait --for=condition=Ready \ dynamographdeployment/solar-open2-250b-vllm-b200-agg-chat --timeout=30m ``` Disaggregated: ```bash kubectl apply -f recipes/solar-open2-250b/vllm/disagg-b200-chat/deploy.yaml kubectl wait --for=condition=Ready \ dynamographdeployment/solar-open2-250b-vllm-b200-disagg-chat --timeout=30m ``` ## Smoke test Port-forward the frontend of the profile you deployed. For the aggregated profile: ```bash kubectl port-forward svc/solar-open2-250b-vllm-b200-agg-chat-frontend 8000:8000 & ``` For the disaggregated profile: ```bash kubectl port-forward svc/solar-open2-250b-vllm-b200-disagg-chat-frontend 8000:8000 & ``` Then, against either: ```bash curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"solar-open2-250b", "messages":[{"role":"user","content":"What is 12 times 7?"}], "max_tokens":128}' ``` Solar Open2 is a reasoning model. Short generations may return the answer in `reasoning_content` with `content` empty; allow a larger `max_tokens` to see a final answer in `content`. ## Benchmark Both performance Jobs read a mooncake trace from the `shared-model-cache` PVC. The trace is not part of the model download, so stage it first. The published figures use `nim_turbo_8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl` (1805 requests, mean ISL 37.4k, mean OSL 1008), which ships in this repository under `recipes/nemotron-3-ultra/perf/traces/`. Copy it in through any running pod that mounts the PVC. Note that the worker pods mount that volume at `/shared-model-cache`, while the benchmark Job mounts the same volume at `/model-cache`, so the file staged below is what `TRACE_FILE` refers to: ```bash TRACE=recipes/nemotron-3-ultra/perf/traces/nim_turbo_8k_1k_70kv_chat_new_noschedule_short_15perc.jsonl # The traces are stored in Git LFS. A plain clone leaves a small pointer file in # place of the data, so fetch the real contents first and check the size: the file # is roughly 900 KB, not a few hundred bytes. git lfs install git lfs pull --include="$TRACE" ls -l "$TRACE" # Whichever profile you deployed: DGD=solar-open2-250b-vllm-b200-agg-chat # or solar-open2-250b-vllm-b200-disagg-chat # The workers mount the cache PVC; the frontend does not. Match the worker # components by name rather than excluding the frontend, so that pods carrying no # component-type label are not selected either. POD=$(kubectl get pods \ -l "nvidia.com/dynamo-graph-deployment-name=$DGD,nvidia.com/dynamo-component-type in (worker,prefill,decode)" \ --field-selector status.phase=Running \ -o jsonpath='{.items[0].metadata.name}') kubectl exec "$POD" -- mkdir -p /shared-model-cache/traces kubectl cp "$TRACE" "$POD:/shared-model-cache/traces/" ``` Set `TRACE_FILE` in the Job to match whichever trace you stage. Then run the Job for the profile under test: ```bash kubectl apply -f recipes/solar-open2-250b/perf/perf.yaml # aggregated kubectl apply -f recipes/solar-open2-250b/perf/perf-disagg.yaml # disaggregated ``` Each Job runs the trace once and writes its summary to `ARTIFACT_DIR`. To run it again, delete the Job first and re-apply; Jobs are immutable, so re-applying over an existing one fails. ## Runtime image Both manifests pin a purpose-built runtime image by digest. It extends the Dynamo 1.4.1 vLLM runtime with the Solar Open2 architecture and parser support and a Dynamo frontend compatibility patch. The changes are Python-only. The stock Dynamo 1.4.1 vLLM runtime does not serve this model — it has no Solar Open2 model definition — so the pinned image is required rather than optional. Pin by digest, not by tag. ## Configuration notes ### Both profiles * **NVFP4 weights, FP8 KV cache.** Both manifests set `--kv-cache-dtype fp8`; the disaggregated profile additionally sets `--calculate-kv-scales`. FP8 halves KV bytes relative to BF16, which roughly doubles the KV pool and raises the realised prefix-cache hit rate. * **MoE backend.** `--moe-backend flashinfer_trtllm` with `--enable-flashinfer-autotune`. * **Attention.** FlashInfer is selected automatically. Note that FlashAttention is not compatible with an FP8 KV cache on this model and will fail at startup if forced. * **Hybrid KV pages.** Solar Open2 has 12 grouped-query attention layers and 36 KDA linear-attention layers. The effective KV page is a hybrid page of roughly 1072 tokens at `--block-size 16`, not the block size itself. Tools that assume block size equals page size need adjusting. * Speculative decoding is not supported on this checkpoint and is not enabled. ### Aggregated profile * Two worker replicas at tensor parallel size 2, fronted by a KV-aware router (`--router-mode kv`, `--router-kv-events`) with a ZMQ KV-event publisher on each worker. KV-aware routing outperforms round-robin on prefix-reuse workloads. * **Expert parallelism is left off.** It measured as neutral for aggregated serving on this checkpoint, so the simpler configuration is used. ### Disaggregated profile * Separate prefill and decode workers connected by `NixlConnector`, with `kv_role: kv_producer` on prefill and `kv_role: kv_consumer` on decode. * **The topology is one prefill worker and two decode workers** — TP4 with expert parallelism for prefill (4 GPUs), TP2 for each decode worker (4 GPUs), 8 GPUs in total. * **`VLLM_SSM_CONV_STATE_LAYOUT=DS` is required.** Solar Open2's KDA layers register as SSM layers, and their convolution state must cross the KV-transfer boundary. Without this variable the engine asserts during initialization and the workers will not start. * **Each worker requests `rdma/ib` equal to its GPU count**, four on prefill and two on each decode worker. This is required, not optional: without it the device plugin does not inject `/dev/infiniband`, UCX falls back to TCP, and nothing logs an error. Verify inside a running worker with `kubectl exec -c main -- ls /dev/infiniband` and expect `uverbsN`, `umadN` and `rdma_cm`. One HCA per GPU is correct; requesting more measured worse. If your RDMA device plugin advertises a different resource name than `rdma/ib`, substitute it in both the requests and the limits. * Transport is configured with `NCCL_IB_DISABLE=0`, `UCX_TLS=^cuda_ipc`, `UCX_RNDV_SCHEME=get_zcopy` and `UCX_MAX_RNDV_RAILS=4`. `UCX_MAX_RNDV_RAILS` tracks the number of HCAs per worker; adjust it if your nodes differ. `UCX_NET_DEVICES` is deliberately left unset; pinning it produced `NIXL_ERR_BACKEND`. * **Per-user output throughput runs close to the floor.** The disaggregated profile clears the 50 tok/s target with little margin, and individual runs on contended hardware may land at or below it. * **The frontend uses round-robin routing, not KV-aware routing.** This is deliberate. KV-aware routing improves the aggregated profile, but with a single prefill worker there is nothing for a prefix-aware router to choose between. KV-aware and load-aware routing were both measured on this topology and neither improved on round-robin. ## Known limitations * The disaggregated profile delivers slightly lower per-GPU throughput than the 4-GPU aggregated profile, at roughly an order of magnitude higher time-to-first-token. Prefer the aggregated profile unless independent prefill/decode scaling is the priority. * The 1M context length requires sufficient KV capacity; reduce `--max-model-len` if deploying on a smaller GPU allocation. * Speculative decoding is not available on this checkpoint. ### API compatibility Some OpenAI-compatible surfaces do not behave as advertised. These originate in the Dynamo frontend, are not specific to this model, and no recipe setting affects them. * **Logprobs.** Requesting `logprobs` on `/v1/chat/completions` returns HTTP 500. * **Responses API.** `store=true` succeeds but the response is not retrievable afterwards (`GET /v1/responses/{id}` returns 404). Unsupported parameters such as `frequency_penalty` are accepted and silently ignored. * **Completions.** `stop_token_ids` with more than three IDs returns HTTP 500 rather than a 400. ### Reasoning budget and structured output The recipe sets `reasoning_effort: high`. Generations are long: on GPQA Diamond the median output is roughly 6,400 tokens, the 99th percentile 131,000, and the longest observed 209,000. A small `max_tokens` therefore truncates the model mid-reasoning and returns `finish_reason=length` with no final content, which shows up most often on structured-output requests where the JSON never arrives in `message.content`. Omit `max_tokens` and let the 1M context window bound the request, or set it well above the reasoning length the workload needs. Values in the low thousands are not sufficient for this model. ## Measured performance Chat workload, 15% subset of the internal `nim_turbo` 8k/1k 70kv trace (1805 requests, mean ISL 37.4k tokens, mean OSL 1008 tokens). SLA: p50 TTFT below 5 s and p50 user output throughput at or above 50 tok/s. | Workload | Recipe | Framework | SKU | Concurrency | System output tok/s/GPU | User output tok/s (P50) | TTFT P50 (ms) | | ----------------- | ---------------------------------------- | --------- | ---- | ----------: | ----------------------: | ----------------------: | ------------: | | Chat (15% subset) | Aggregated, 2 replicas, KV-aware routing | vLLM | B200 | 20 | 216.88 | 52.02 | 308 | | Chat (15% subset) | Disaggregated 1P:2D, round-robin | vLLM | B200 | 44 | 206.20 | 51.69 | 2277 | Figures are measured with the profile's benchmark manifest (`perf.yaml` for aggregated, `perf-disagg.yaml` for disaggregated). That configuration isolates the warmup phase from profiling; it does not clear server-side KV cache. For runs that must start from a cold cache, restart the worker pods between them. ## Source * Manifests: `recipes/solar-open2-250b/` * Model: `nota-ai/Solar-Open2-250B-Nota-NVFP4`