Parallel Media Decoding

Parallel media decoding moves media fetching and decoding from the inference backend to the NVIDIA Dynamo Rust frontend. The frontend decodes supported images and videos on a CPU worker pool and transfers the decoded pixel buffers to the backend through NIXL.

The backend still runs its model-specific multimodal processor and vision encoder. This feature changes where media input is decoded; it does not skip vision encoding.

Support Matrix

Agg refers to an aggregated worker. The entries in this matrix represent the supported topologies for frontend decoding.

SGLang frontend video decoding is available only on CUDA. SGLang XPU images do not include the in-tree FFmpeg decoder. Image frontend decoding is not subject to that codec restriction.

This matrix describes parallel media decoding, not the overall multimodal support of each backend. A backend can support video or audio by decoding it on the worker even when the frontend decoding path does not support that modality.

When to Use

Use parallel media decoding when media preprocessing consumes a significant part of request latency or backend CPU time. It is most useful for workloads with:

• Concurrent requests containing HTTP, HTTPS, or base64-encoded media
• Multiple images or supported videos in one request
• Backend workers whose request path is constrained by media fetching or decompression

Parallel media decoding can also be combined with the embedding cache. Frontend decoding reduces media input processing work, while the embedding cache can skip vision encoding for repeated images.

How It Works

For each request, the frontend:

1. Fetches the media URL or decodes the base64 data URL.
2. Decodes supported images or VP8/VP9 videos on a CPU worker pool.
3. Registers the decoded pixel buffer with NIXL.
4. Sends the buffer descriptor to the selected backend worker.

The backend reads the decoded pixels through NIXL, then continues with its normal multimodal processor and vision encoder.

Enable Parallel Media Decoding

Add --frontend-decoding to the backend worker command. Do not add the flag to dynamo.frontend; the backend advertises the decoder configuration when it registers the model.

JPEG Decoder

The Rust frontend uses libjpeg-turbo for JPEG inputs by default. To use image::ImageReader instead, set DYN_MM_ENABLE_LIBJPEG=0 on the frontend process. Dynamo backend runtime images include libturbojpeg; custom images must provide libturbojpeg.so.0. If the library is unavailable, Dynamo logs a one-time warning and falls back to image::ImageReader.

NIXL Progress Thread

For each worker set with frontend decoding, the frontend creates up to two NIXL agents: one for chat requests and one for completions requests. The progress thread of each agent sleeps for at most DYN_MM_NIXL_PROGRESS_DELAY_US microseconds (default 1000, range 0 to 1000000), rounded up to whole milliseconds, and wakes early on UCX events.

When the frontend and the backend worker cannot share memory or use RDMA (for example, the frontend runs in its own pod with no RDMA device), UCX falls back to TCP. Over TCP, some UCX events do not wake the thread, so a read can wait up to the full delay. In a same-host TCP test with UCX 1.22, the default delay added about 0.7 ms to each 3 MiB read and 3 ms to each 12 MiB read. Larger values make each wait longer. Set DYN_MM_NIXL_PROGRESS_DELAY_US=0 on the frontend process to trade one CPU core for each NIXL agent for that latency.

Requirements and Limitations

• The published nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0 image installs NIXL wheels but does not expose the native NIXL and UCX libraries and plugins required by the Rust frontend. It cannot run parallel media decoding as shipped. Run the frontend in a backend runtime image or environment where NIXL and UCX are configured and available.