> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.6.1) (default): https://docs.dynamo.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.dynamo.nvidia.com/dynamo/dev/llms.txt > - v1.6.1: https://docs.dynamo.nvidia.com/dynamo/v1.6.1/llms.txt > - v1.6.0: https://docs.dynamo.nvidia.com/dynamo/v1.6.0/llms.txt > - v1.5.1: https://docs.dynamo.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.dynamo.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.dynamo.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.dynamo.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.dynamo.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.dynamo.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.dynamo.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.dynamo.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.dynamo.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.dynamo.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.dynamo.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.dynamo.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.dynamo.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.dynamo.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.dynamo.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.dynamo.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.dynamo.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.dynamo.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.dynamo.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Video Understanding Dynamo serves video-capable vision language models through the OpenAI-compatible `/v1/chat/completions` endpoint. A request carries a `video_url` content part next to its text. Dynamo decodes the clip, samples frames, and the worker runs the model's vision encoder on them. This page gets one video request working on each backend. To generate video instead, see [Text to Video](/dynamo/dev/diffusion/text-to-video). To route repeated clips to the worker that already holds their KV cache, see [Exact Video KV Routing](/dynamo/dev/video-understanding/exact-video-kv-routing). ## Support Matrix Which codecs work depends on where the video is decoded: | Backend | H.264 / H.265 | VP8 / VP9 | Notes | | --- | --- | --- | --- | | vLLM | Yes (NVDEC on the GPU) | Yes, with `--frontend-decoding` | | | SGLang | Disaggregated Qwen2-family only (NVDEC) | Yes, aggregated with `--frontend-decoding` on CUDA | Aggregated VP8/VP9 is validated for Qwen3-VL and Qwen3.5 | | TensorRT-LLM | Yes (NVDEC on the GPU) | Through the TensorRT-LLM loader, after you [install a software decoder](/dynamo/dev/image-understanding/additional-media-decoders) | | - **NVDEC** is the GPU's hardware video decoder. The container must be granted the `video` driver capability, or hardware decode is silently unavailable. - **Frontend decoding** (`--frontend-decoding` on the worker) decodes VP8 and VP9 on the CPU in Dynamo's Rust frontend. It does not decode H.264, H.265, or AV1. - The runtime images ship no software H.264, H.265, or AV1 decoder. See [Video Decoding and GPU Requirements](/dynamo/dev/video-understanding/video-decoding) for GPU support, container setup, and installing additional decoders. ## Set up the backend The vLLM and TensorRT-LLM launchers below start the worker with `--enable-multimodal`. The SGLang aggregated launcher (`agg_vision.sh`) does not pass it. Run the commands from the root of a Dynamo repository checkout. Each launcher starts a frontend on port 8000 and one aggregated worker. Requires a working [vLLM backend setup](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/overview). ```bash bash examples/backends/vllm/launch/agg_multimodal.sh --model Qwen/Qwen3-VL-2B-Instruct ``` To accept VP8 and VP9 input, append `--frontend-decoding`; the launcher passes extra arguments through to the worker: ```bash bash examples/backends/vllm/launch/agg_multimodal.sh --model Qwen/Qwen3-VL-2B-Instruct --frontend-decoding ``` Requires a working [SGLang backend setup](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/overview). ```bash bash examples/backends/sglang/launch/agg_vision.sh --model-path Qwen/Qwen3-VL-2B-Instruct --frontend-decoding ``` Aggregated video on SGLang requires `--frontend-decoding` and a CUDA device. SGLang XPU images omit the frontend decoder. For H.264 or H.265 on Qwen2-family models, use the disaggregated encode-worker flow in [SGLang Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal). Requires a working [TensorRT-LLM backend setup](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/overview). ```bash bash examples/backends/trtllm/launch/agg_multimodal.sh ``` The launcher serves `Qwen/Qwen2-VL-7B-Instruct`. Set `"model": "Qwen/Qwen2-VL-7B-Instruct"` in the request below. See [TensorRT-LLM Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal) for other models. TensorRT-LLM has no frontend video decoding, and the runtime image ships no software VP8/VP9 decoder. The example clip below is VP8/VP9, so on TensorRT-LLM it needs a software decoder installed as described in [Install Additional Decoders](/dynamo/dev/video-understanding/video-decoding#install-additional-decoders), or an H.264 or H.265 clip of your own. ## Send a Video Request In another terminal, send a chat request with a `video_url` content part. This example clip is VP8/VP9 WebM, so it needs `--frontend-decoding` on vLLM and SGLang, or a separately installed software decoder on TensorRT-LLM: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-VL-2B-Instruct", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this video."}, {"type": "video_url", "video_url": {"url": "https://interactive-examples.mdn.mozilla.net/media/cc0-videos/flower.webm"}} ] }], "max_tokens": 64 }' ``` `DYN_MM_VIDEO_NUM_FRAMES` (default `32`) sets the maximum number of frames sampled from each clip. Keep it the same on the frontend and every worker. Video URLs are fetched under the same default-deny policy as images: `https://` and `data:` sources pass, and internal or plain `http://` hosts need `DYN_MM_ALLOW_INTERNAL=1`. See [Image Understanding › Media URL Policy](/dynamo/dev/image-understanding/overview#media-url-policy). ## Next Steps - [Exact Video KV Routing](/dynamo/dev/video-understanding/exact-video-kv-routing): reuse KV cache for repeated clips across workers. - [Video Decoding and GPU Requirements](/dynamo/dev/video-understanding/video-decoding): NVDEC, container capabilities, and extra decoders. ## Performance Optimizations Video requests share most of their performance features with image requests. [VLM Performance Optimization](/dynamo/dev/image-understanding/vlm-performance-optimization) shows where a request spends its time and when each feature helps. The ones that apply to video: - [Frontend media decoding](/dynamo/dev/image-understanding/vlm-performance-optimization/frontend-media-decoding): decode VP8/VP9 video in the Rust frontend. - [Embedding cache](/dynamo/dev/image-understanding/vlm-performance-optimization/embedding-cache): skip the vision encoder for clips it has seen. - [Multimodal KV routing](/dynamo/dev/image-understanding/vlm-performance-optimization/multimodal-kv-routing) and [Exact Video KV Routing](/dynamo/dev/video-understanding/exact-video-kv-routing): send repeated clips to the worker that already holds their KV cache.