> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.nvidia.com/dynamo/dev/llms.txt
> - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Parallel Media Decoding

Parallel media decoding moves media fetching and decoding from the inference
backend to the NVIDIA Dynamo Rust frontend. The frontend decodes supported
images and videos on a CPU worker pool and transfers the decoded pixel buffers
to the backend through NIXL.

The backend still runs its model-specific multimodal processor and vision
encoder. This feature changes where media input is decoded; it does not skip
vision encoding.

## Support Matrix

| Input modality | vLLM | SGLang | TensorRT-LLM |
| --- | --- | --- | --- |
| Image | Agg | Agg | Agg |
| Video | Agg (VP8/VP9) | Agg (VP8/VP9) | Not supported |
| Audio | Not supported | Not supported | Not supported |

`Agg` refers to an aggregated worker. The entries in this matrix represent the
supported topologies for frontend decoding.

SGLang frontend video decoding is available only on CUDA. SGLang XPU images do
not include the in-tree FFmpeg decoder. Image frontend decoding is not subject
to that codec restriction.

This matrix describes parallel media decoding, not the overall multimodal
support of each backend. A backend can support video or audio by decoding it on
the worker even when the frontend decoding path does not support that modality.

## When to Use

Use parallel media decoding when media preprocessing consumes a significant
part of request latency or backend CPU time. It is most useful for workloads
with:

- Concurrent requests containing HTTP, HTTPS, or base64-encoded media
- Multiple images or supported videos in one request
- Backend workers whose request path is constrained by media fetching or
  decompression

Parallel media decoding can also be combined with the [embedding
cache](/dynamo/dev/multimodal/embedding-cache). Frontend decoding reduces media input processing
work, while the embedding cache can skip vision encoding for repeated images.

## How It Works

For each request, the frontend:

1. Fetches the media URL or decodes the base64 data URL.
2. Decodes supported images or VP8/VP9 videos on a CPU worker pool.
3. Registers the decoded pixel buffer with NIXL.
4. Sends the buffer descriptor to the selected backend worker.

The backend reads the decoded pixels through NIXL, then continues with its
normal multimodal processor and vision encoder.

## Enable Parallel Media Decoding

Add `--frontend-decoding` to the backend worker command. Do not add the flag to
`dynamo.frontend`; the backend advertises the decoder configuration when it
registers the model.

| Backend | Worker flags |
| --- | --- |
| [vLLM](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/reference-guide) | `--enable-multimodal --frontend-decoding` |
| [SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/reference-guide) | `--frontend-decoding` |
| [TensorRT-LLM](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/reference-guide) | `--enable-multimodal --frontend-decoding` |

### JPEG Decoder

The Rust frontend uses libjpeg-turbo for JPEG inputs by default. To use
`image::ImageReader` instead, set `DYN_MM_ENABLE_LIBJPEG=0` on the frontend
process. Dynamo backend runtime images include `libturbojpeg`; custom images
must provide `libturbojpeg.so.0`. If the library is unavailable, Dynamo logs a
one-time warning and falls back to `image::ImageReader`.

### NIXL Progress Thread

For each worker set with frontend decoding, the frontend creates up to two NIXL
agents: one for chat requests and one for completions requests. The progress
thread of each agent sleeps for at most `DYN_MM_NIXL_PROGRESS_DELAY_US`
microseconds (default `1000`, range `0` to `1000000`), rounded up to whole
milliseconds, and wakes early on UCX events.

When the frontend and the backend worker cannot share memory or use RDMA (for
example, the frontend runs in its own pod with no RDMA device), UCX falls back
to TCP. Over TCP, some UCX events do not wake the thread, so a read can wait up
to the full delay. In a same-host TCP test with UCX 1.22, the default delay
added about 0.7 ms to each 3 MiB read and 3 ms to each 12 MiB read. Larger
values make each wait longer. Set `DYN_MM_NIXL_PROGRESS_DELAY_US=0` on the
frontend process to trade one CPU core for each NIXL agent for that latency.

## Requirements and Limitations

- The published `nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0` image installs
  NIXL wheels but does not expose the native NIXL and UCX libraries and plugins
  required by the Rust frontend. It cannot run parallel media decoding as
  shipped. Run the frontend in a backend runtime image or environment where
  NIXL and UCX are configured and available.