vLLM-Omni

Dynamo supports multimodal generation through the vLLM-Omni backend. This integration exposes text-to-text, text-to-image, and text-to-video capabilities via OpenAI-compatible API endpoints.

Prerequisites

This guide assumes familiarity with deploying Dynamo with vLLM as described in the vLLM backend guide.

Installation

Dynamo container images include vLLM-Omni pre-installed. If you are using pip install ai-dynamo[vllm], vLLM-Omni is not included automatically because the matching release is not yet available on PyPI. Install it separately from source:

$
pip install git+https://github.com/vllm-project/vllm-omni.git@v0.16.0rc1

Supported Modalities

The --output-modalities flag determines which endpoint(s) the worker registers. When set to image, both /v1/chat/completions (returns inline base64 images) and /v1/images/generations are available. When set to video, the worker serves /v1/videos.

Tested Models

To run a non-default model, pass --model to any launch script:

$
bash examples/backends/vllm/launch/agg_omni_image.sh --model AIDC-AI/Ovis-Image-7B
$
bash examples/backends/vllm/launch/agg_omni_video.sh --model Wan-AI/Wan2.2-T2V-A14B-Diffusers

Text-to-Text

Launch an aggregated deployment (frontend + omni worker):

$
bash examples/backends/vllm/launch/agg_omni.sh

This starts Qwen/Qwen2.5-Omni-7B with a single-stage thinker config on one GPU.

Verify the deployment:

$
curl -s http://localhost:8000/v1/chat/completions \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen2.5-Omni-7B",
>
    "messages": [{"role": "user", "content": "What is 2+2?"}],
>
    "max_tokens": 50,
>
    "stream": false
>
  }'

This script uses a custom stage config (stage_configs/single_stage_llm.yaml) that configures the thinker stage for text generation. See Stage Configuration for details.

Text-to-Image

Launch using the provided script with Qwen/Qwen-Image:

$
bash examples/backends/vllm/launch/agg_omni_image.sh

Via /v1/chat/completions

$
curl -s http://localhost:8000/v1/chat/completions \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen-Image",
>
    "messages": [{"role": "user", "content": "A cat sitting on a windowsill"}],
>
    "stream": false
>
  }'

The response includes base64-encoded images inline:

1
{
2
  "choices": [{
3
    "delta": {
4
      "content": [
5
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
6
      ]
7
    }
8
  }]
9
}

Via /v1/images/generations

$
curl -s http://localhost:8000/v1/images/generations \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen-Image",
>
    "prompt": "A cat sitting on a windowsill",
>
    "size": "1024x1024",
>
    "response_format": "url"
>
  }'

Text-to-Video

Launch using the provided script with Wan-AI/Wan2.1-T2V-1.3B-Diffusers:

$
bash examples/backends/vllm/launch/agg_omni_video.sh

Generate a video via /v1/videos:

$
curl -s http://localhost:8000/v1/videos \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
>
    "prompt": "A drone flyover of a mountain landscape",
>
    "seconds": 2,
>
    "size": "832x480",
>
    "response_format": "url"
>
  }'

The response returns a video URL or base64 data depending on response_format:

1
{
2
  "id": "...",
3
  "object": "video",
4
  "model": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
5
  "status": "completed",
6
  "data": [{"url": "file:///tmp/dynamo_media/videos/req-abc123.mp4"}]
7
}

The /v1/videos endpoint also accepts NVIDIA extensions via the nvext field for fine-grained control:

CLI Reference

For the full list of Omni-related flags (including --omni, --output-modalities, --stage-configs-path, --media-output-fs-url, --media-output-http-url, and the --omni-* diffusion flags), run:

$
python -m dynamo.vllm --help

See also the Argument Reference in the Reference Guide.

Storage Configuration

Generated images and videos are stored via fsspec, which supports local filesystems, S3, GCS, and Azure Blob.

By default, media is written to the local filesystem at file:///tmp/dynamo_media. To use cloud storage:

$
bash examples/backends/vllm/launch/agg_omni_video.sh \
>
  --media-output-fs-url s3://my-bucket/media \
>
  --media-output-http-url https://cdn.example.com/media

When --media-output-http-url is set, response URLs are rewritten as {base-url}/{storage-path} (e.g., https://cdn.example.com/media/videos/req-id.mp4). When unset, the raw filesystem path is returned.

For S3 credential configuration, set the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) or use IAM roles. See the fsspec S3 docs for details.

Stage Configuration

Omni pipelines are configured via YAML stage configs. See examples/backends/vllm/launch/stage_configs/single_stage_llm.yaml for an example. For full documentation on stage config format and multi-stage pipelines, refer to the vLLM-Omni Stage Configs documentation.

Current Limitations

• Only text prompts are supported as input (no multimodal input yet).
• KV cache events are not published for omni workers.
• Each worker supports a single output modality at a time.