> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# vLLM

Dynamo vLLM integrates [vLLM](https://github.com/vllm-project/vllm) engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, and request cancellation while maintaining full compatibility with vLLM's native engine arguments. Dynamo leverages vLLM's native KV cache events, NIXL-based transfer mechanisms, and metric reporting to enable KV-aware routing and P/D disaggregation.

## Experimental Sidecar

The experimental sidecar path runs the Dynamo worker outside vLLM and connects
through vLLM's native gRPC API. It keeps vLLM's native server and argument
surface while separating Dynamo and engine dependencies. See
[vLLM Sidecar](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/sidecar) for current readiness and launch examples.

## Installation

### Install Latest Release

We recommend using [uv](https://github.com/astral-sh/uv) to install:

```bash
uv venv --python 3.12 --seed
uv pip install "ai-dynamo[vllm]"
```

This installs Dynamo with the compatible vLLM version.

---

### Container

We have public images available on [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts):

```bash
docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version>
./container/run.sh -it --framework VLLM --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version>
```

<Accordion title="Build from source">

```bash
python container/render.py --framework vllm --output-short-filename
docker build -f container/rendered.Dockerfile -t dynamo:latest-vllm .
```

```bash
./container/run.sh -it --framework VLLM [--mount-workspace]
```

</Accordion>

### Development Setup

For development, use the [devcontainer](https://github.com/ai-dynamo/dynamo/tree/main/.devcontainer) which has all dependencies pre-installed.

## Feature Support Matrix

| Feature | Status | Notes |
|---------|--------|-------|
| [**Disaggregated Serving**](/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving) | ✅ | Prefill/decode separation with NIXL KV transfer |
| [**KV-Aware Routing**](/dynamo/dev/knowledge-base/modular-components/router/overview) | ✅ | |
| [**SLA-Based Planner**](/dynamo/dev/knowledge-base/modular-components/planner/planner-guide) | ✅ | |
| [**KVBM**](/dynamo/dev/knowledge-base/modular-components/kvbm/overview) | ✅ | |
| [**LMCache**](/dynamo/dev/cli/kv-cache-offloading/overview) | ✅ | CUDA 12.9 and arm64/aarch64 containers may require building LMCache from source |
| [**FlexKV**](/dynamo/dev/cli/kv-cache-offloading/overview) | ✅ | Requires a separate FlexKV build |
| [**Multimodal Support**](/dynamo/dev/diffusion/overview) | ✅ | Via vLLM-Omni integration |
| [**Observability**](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/observability) | ✅ | Metrics and monitoring |
| **WideEP** | ✅ | Support for DeepEP |
| **DP Rank Routing** | ✅ | [Hybrid load balancing](https://docs.vllm.ai/en/stable/serving/data_parallel_deployment/?h=external+dp#hybrid-load-balancing) via external DP rank control |
| [**LoRA**](https://github.com/ai-dynamo/dynamo/tree/main/examples/backends/vllm/launch/lora/README.md) | ✅ | Dynamic loading/unloading from S3-compatible storage |
| **GB200 Support** | ✅ | Container functional on main |

## Feature Interactions

vLLM offers the broadest feature coverage in Dynamo, with full support for disaggregated serving, KV-aware routing, KV block management, LoRA adapters, and multimodal inference including video and audio. The matrix below shows which feature pairs are validated to work together.

**Legend:** ✅ Supported &nbsp;|&nbsp; 🚧 Work in Progress / Experimental / Limited

| Feature | Disaggregated Serving | KV-Aware Routing | SLA-Based Planner | KV Block Manager | Multimodal | Request Migration | Request Cancellation | LoRA | Tool Calling | Speculative Decoding |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **Disaggregated Serving** | — | | | | | | | | | |
| **KV-Aware Routing** | ✅ | — | | | | | | | | |
| **SLA-Based Planner** | ✅ | ✅ | — | | | | | | | |
| **KV Block Manager** | ✅ | ✅ | ✅ | — | | | | | | |
| **Multimodal** | ✅ | ✅<sup>1</sup> | — | ✅ | — | | | | | |
| **Request Migration** | ✅ | ✅ | ✅ | ✅ | ✅ | — | | | | |
| **Request Cancellation** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | | | |
| **LoRA** | ✅ | ✅<sup>2</sup> | — | ✅ | — | ✅ | ✅ | — | | |
| **Tool Calling** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | |
| **Speculative Decoding** | ✅ | ✅ | — | ✅ | — | ✅ | ✅ | — | ✅ | — |

> **Notes:**
> 1. **Multimodal + KV-Aware Routing**: Image-aware KV routing is supported in the documented vLLM paths. The default Rust frontend path supports model families handled by `llm-multimodal`; the Python chat-processor path delegates to vLLM's multimodal processor. ([Source](/dynamo/dev/multimodal/multimodal-kv-routing))
> 2. **KV-Aware LoRA Routing**: vLLM supports routing requests based on LoRA adapter affinity.
> 3. **Audio Support**: vLLM supports audio models like Qwen2-Audio (experimental). ([Source](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal))
> 4. **Video Support**: vLLM supports video input with frame sampling. ([Source](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal))
> 5. **Speculative Decoding**: Eagle3 support documented. ([Source](/dynamo/dev/additional-resources/speculative-decoding/speculative-decoding-with-v-llm))

## Quick Start

Start infrastructure services for local development:

```bash
docker compose -f dev/docker-compose.yml up -d
```

Launch an aggregated serving deployment:

```bash
cd $DYNAMO_HOME/examples/backends/vllm
bash launch/agg.sh
```

> **Running launch scripts standalone.** The `launch/*.sh` scripts expect etcd and NATS to be reachable on localhost. Bring them up first (run from the repo root, or use the absolute path shown):
>
> ```bash
> docker compose -f "$DYNAMO_HOME/dev/docker-compose.yml" up -d
> ```
>
> Then run the launch script. Without these, workers register but the frontend cannot discover them and requests hang.

## Next Steps

- **[Reference Guide](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/reference-guide)**: Configuration, arguments, and operational details
- **[Examples](/dynamo/dev/recipes/cli-templates/v-llm)**: Local deployment launch scripts
- **[KV Cache Offloading](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/kv-cache-offloading)**: KVBM, LMCache, and FlexKV integrations
- **[Observability](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/observability)**: Metrics and monitoring
- **[vLLM-Omni](/dynamo/dev/diffusion/overview)**: Multimodal model serving
- **[Kubernetes Deployment](https://github.com/ai-dynamo/dynamo/tree/main/examples/backends/vllm/deploy/README.md)**: Kubernetes deployment guide
- **[vLLM Documentation](https://docs.vllm.ai/en/stable/)**: Upstream vLLM serve arguments