Decision Models with SGLang
Run a dedicated text-scoring worker for both Dynamo decision APIs
Experimental. Run an aggregated SGLang CUDA worker to serve OpenAI POST /v1/decisions and Jev POST /v1/systemone through NVIDIA Dynamo. Both routes use the same candidate-scoring implementation. They do not expose SGLang’s native Decisions wire schema.
Prerequisites
Use a Dynamo environment built with SGLang and the services described in Local Installation. Use a supported GPU and ensure the checkpoint fits available memory. The example uses Qwen/Qwen3-0.6B as a small integration-test target, not a claim of decision-task quality or qualification of larger Qwen models.
The worker capability gate accepts SGLang 0.5.19, 0.5.22, and 0.5.23, including their recognized CUDA package suffixes. The current CUDA container pins 0.5.23.
Decision scoring requires --max-running-requests 1 --disable-overlap-schedule. The worker processes one inference request at a time, including any chat or completion requests sent to it. Dynamo does not advertise decision scoring if either scheduling requirement is violated. Use a separate worker for throughput-oriented chat traffic to avoid this serialization; separation is a serving recommendation, while the scheduling settings are requirements.
Decision capability publication also requires an aggregated token-input worker, one data-parallel rank, and pipeline parallelism of one, with attention data parallelism, speculative decoding, diffusion execution, and priority preemption disabled. These are enforced eligibility checks, not tuning suggestions. CUDA is the qualified device for this guide; other devices and LoRA decision execution are not qualified here.
Start the Frontend
In the first terminal, select a dedicated namespace and enable both public routes:
--enable-systemone-api defaults to false and enables both public decision APIs.
Start the Worker
In a second terminal using the same Dynamo environment and runtime services:
Wait for the Published SGLang ... System One capability message before sending requests. Ordinary generation can work even when the worker does not meet the decision eligibility requirements above.
Verify Both APIs
List the model and check its serving readiness:
Run the curl, OpenAI SDK, and Jev SDK examples against this frontend. Use the exact served model alias. Both endpoints return question answers and usage; this SGLang profile must report output_tokens: 0 because it requests candidate log-probabilities without generating a token.
Profile Limits
The protocol ceiling is not a promise that this tokenizer can represent 255 distinct labels. Unsupported candidate sets fail before inference. Input-token usage sums the rendered prompts for all questions; cache reads use measured backend counters. See Decision API Configuration for independent byte, token-work, admission, and deadline limits.
The small-model SGLang numerical qualification fixture compares Dynamo with an independent native scorer. A passing endpoint smoke test does not replace that numerical check, task-specific accuracy evaluation, or distributed lifecycle testing. Engine-version acceptance alone is not evidence that a model was qualified.
Stop the Example
Stop the worker and frontend with Ctrl-C in their respective terminals. Keep shared services running if another deployment uses them.