跳到导航
Knowledge BaseModular ComponentsBackendsSGLang

SGLang Sidecar

Run Dynamo beside a stock SGLang engine through native gRPC.
以 Markdown 格式查看

Experimental. The sidecars and their deployment examples are experimental. Manifests, flags, and behavior may change without notice.

dynamo-sglang-sidecar connects a Dynamo worker to SGLang’s native gRPC server. See Sidecar Backends for the architecture.

For the best and latest support, use the upstream SGLang nightly image, which carries the latest gRPC server updates: lmsysorg/sglang:dev.

Support Matrix

FeatureSupported
AggregatedYes
DisaggregatedYes
KV routingYes

Launch Locally

See lib/sidecar/sglang/launch/ for all topologies. For example, aggregated serving on one GPU:

export DYN_DISCOVERY_BACKEND=file   # single host: no etcd or NATS needed
./lib/sidecar/sglang/launch/agg.sh

In a second terminal:

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Deploy on Kubernetes

See lib/sidecar/sglang/deploy/ for all manifests. For example, aggregated serving:

Before applying, replace <your-registry>/dynamo-sidecar in the manifest with a sidecar image and create the hf-token-secret Secret that the manifest reads.

kubectl create secret generic hf-token-secret -n <namespace> \
  --from-literal=HF_TOKEN=<your-hf-token>
kubectl apply -f lib/sidecar/sglang/deploy/agg.yaml -n <namespace>
kubectl port-forward -n <namespace> svc/sglang-sidecar-agg-frontend 8000:8000

Topologies

The frontend reaches each sidecar over Dynamo’s request, discovery, and event planes; the sidecar reaches the engine over its native gRPC API. Dashed arrows carry KV events.

Single-Node TP

One engine on one node, with one sidecar.

On one node, a request reaches the SGLang tensor-parallel ranks through the Dynamo Sidecar. The Dynamo Frontend sends requests over the request plane to the sidecar.

Multi-Node TP

One engine spans two nodes. Only the leader node has a sidecar; the follower node holds the remaining TP ranks.

When one SGLang engine spans two nodes with tensor parallelism, only the leader node runs a Dynamo Sidecar. The Dynamo Frontend sends requests over the request plane to the sidecar on Node 0.

Multi-Node DP

The leader sidecar registers every DP rank and serves all requests; SGLang dispatches each one to the right scheduler. The follower sidecar accepts no requests and relays its node’s KV events directly to the frontend.

SGLang data parallelism across two nodes. Only the leader sidecar serves requests: the Dynamo Frontend router picks a DP rank and sends requests over the request plane to the node 0 Dynamo Sidecar, which registers DP ranks 0-3 and calls its local SGLang over native gRPC.