Running SGLang with Dynamo

Use the Latest Release

We recommend using the latest stable release of dynamo to avoid breaking changes:

You can find the latest release here and check out the corresponding branch with:

$
git checkout $(git describe --tags $(git rev-list --tags --max-count=1))

Table of Contents

• Feature Support Matrix
• Dynamo SGLang Integration
• Installation
• Quick Start
• Single Node Examples
• Multi-Node and Advanced Examples
• Deploy on SLURM or Kubernetes

Feature Support Matrix

Core Dynamo Features

Dynamo SGLang Integration

Dynamo SGLang integrates SGLang engines into Dynamo’s distributed runtime, enabling advanced features like disaggregated serving, KV-aware routing, and request migration while maintaining full compatibility with SGLang’s engine arguments.

Argument Handling

Dynamo SGLang uses SGLang’s native argument parser, so most SGLang engine arguments work identically. You can pass any SGLang argument (like --model-path, --tp, --trust-remote-code) directly to dynamo.sglang.

Dynamo-Specific Arguments

Tokenizer Behavior

• Default (--use-sglang-tokenizer not set): Dynamo handles tokenization/detokenization via our blazing fast frontend and passes input_ids to SGLang
• With --use-sglang-tokenizer: SGLang handles tokenization/detokenization, Dynamo passes raw prompts

Request Cancellation

When a user cancels a request (e.g., by disconnecting from the frontend), the request is automatically cancelled across all workers, freeing compute resources for other requests.

Cancellation Support Matrix

For more details, see the Request Cancellation Architecture documentation.

Installation

Install latest release

We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with curl -LsSf https://astral.sh/uv/install.sh | sh

$
# create a virtual env
$
uv venv --python 3.12 --seed
$
# install the latest release (which comes bundled with a stable sglang version)
$
uv pip install "ai-dynamo[sglang]"

Install editable version for development

$
# create a virtual env
$
uv venv --python 3.12 --seed
$
# build dynamo runtime bindings
$
uv pip install maturin
$
cd $DYNAMO_HOME/lib/bindings/python
$
maturin develop --uv
$
cd $DYNAMO_HOME
$
# installs sglang supported version along with dynamo
$
# include the prerelease flag to install flashinfer rc versions
$
uv pip install -e .
$
# install any sglang version >= 0.5.3.post2
$
uv pip install "sglang[all]==0.5.3.post2"

Using docker containers

$
cd $DYNAMO_ROOT
$
./container/build.sh \
>
  --framework SGLANG \
>
  --tag dynamo-sglang:latest \

$
docker run \
>
    --gpus all \
>
    -it \
>
    --rm \
>
    --network host \
>
    --shm-size=10G \
>
    --ulimit memlock=-1 \
>
    --ulimit stack=67108864 \
>
    --ulimit nofile=65536:65536 \
>
    --cap-add CAP_SYS_PTRACE \
>
    --ipc host \
>
    dynamo-sglang:latest

Quick Start

Below we provide a guide that lets you run all of our common deployment patterns on a single node.

Start NATS and ETCD in the background

Start using Docker Compose

$
docker compose -f deploy/docker-compose.yml up -d

Aggregated Serving

$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/agg.sh

Aggregated Serving with KV Routing

$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/agg_router.sh

Aggregated Serving for Embedding Models

Here’s an example that uses the Qwen/Qwen3-Embedding-4B model.

$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/agg_embed.sh

$
curl localhost:8000/v1/embeddings \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen3-Embedding-4B",
>
    "input": "Hello, world!"
>
  }'

Disaggregated serving

See SGLang Disaggregation to learn more about how sglang and dynamo handle disaggregated serving.

$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/disagg.sh

Disaggregated Serving with KV Aware Prefill Routing

$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/disagg_router.sh

Disaggregated Serving with Mixture-of-Experts (MoE) models and DP attention

You can use this configuration to test out disaggregated serving with dp attention and expert parallelism on a single node before scaling to the full DeepSeek-R1 model across multiple nodes.

$
# note this will require 4 GPUs
$
cd $DYNAMO_HOME/examples/backends/sglang
$
./launch/disagg_dp_attn.sh

Testing the Deployment

Send a test request to verify your deployment:

$
curl localhost:8000/v1/chat/completions \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "messages": [
>
    {
>
        "role": "user",
>
        "content": "Explain why Roger Federer is considered one of the greatest tennis players of all time"
>
    }
>
    ],
>
    "stream": true,
>
    "max_tokens": 30
>
  }'

Advanced Examples

Below we provide a selected list of advanced examples. Please open up an issue if you’d like to see a specific example!

Run a multi-node sized model

• Run a multi-node model

Large scale P/D disaggregation with WideEP

• Run DeepSeek-R1-FP8 on H100s
• Run DeepSeek-R1-FP8 on GB200s

Hierarchical Cache (HiCache)

• Enable SGLang Hierarchical Cache (HiCache)

Multimodal Encode-Prefill-Decode (EPD) Disaggregation with NIXL

• Run a multimodal model with EPD Disaggregation

Deployment

We currently provide deployment examples for Kubernetes and SLURM.

Kubernetes

• Deploying Dynamo with SGLang on Kubernetes

SLURM

• Deploying Dynamo with SGLang on SLURM