Skip to navigation
Image UnderstandingVLM Performance Optimization

Encoder Disaggregation

Scale vision encoders separately from LLM workers
View as Markdown

In aggregated serving, one worker does media decode, preprocessing, the vision encoder, prefill, and decode. In E/PD disaggregated serving, encoder workers do media decode, preprocessing, and the vision encoder, and send embeddings over NIXL to PD workers that run prefill and decode. A prefill worker checks the embedding cache first and calls the encoder workers only on a miss.

Overview

Encoder disaggregation separates the vision encoder from the prefill/decode pipeline into its own worker. Instead of running image encoding inline, a dedicated encode worker handles media processing and transfers the resulting embeddings to downstream workers via NIXL (RDMA).

This enables:

  • Independent scaling of encode workers based on vision workload
  • Reduced GPU memory pressure on prefill/decode workers
  • Better GPU utilization by matching worker counts to actual bottlenecks

When to Use

Use encoder disaggregation when:

  • Vision encoding is a bottleneck and you need to scale encoders independently of LLM workers
  • You want to run the vision encoder on different hardware (e.g., smaller GPUs for encoding, larger GPUs for LLM inference)
  • Your deployment handles high volumes of multimodal requests and encoding throughput is limiting

For simple deployments or development and testing, the aggregated pattern (EPD, one worker) is easier to set up.

Deployment Patterns

E/PD

Separate encoder, combined prefill and decode

Frontend → Processor → Encode Worker → PD Worker → Response

The encode worker runs the vision model and transfers embeddings over NIXL to the combined prefill and decode worker.

E/P/D

Separate encoder, prefill, and decode

Frontend → Processor → Encode Worker → Prefill Worker → Decode Worker → Response

The encode worker transfers embeddings over NIXL to the prefill worker. The prefill worker then transfers the KV cache to the decode worker.

Placement Options

Where the encoder runs is a separate choice from whether it is disaggregated:

  • Aggregated: every GPU runs one worker that encodes, prefills, and decodes. This is the simplest setup.
  • Colocated encoder: every GPU runs encoder workers and a prefill and decode (PD) worker side by side, sharing the GPU through CUDA MPS. Encoding no longer queues behind prefill on the same scheduler, and you need no extra hardware.
  • Disaggregated encoder: encoders run on their own GPUs, which can be lower-tier hardware, and send embeddings to the PD workers over NIXL.

Three topologies. Aggregated serving runs an aggregated worker on each of four identical GPUs. Colocated encoder runs encoder workers and a PD worker on each of four identical GPUs using MPS. Disaggregated encoder runs encoder workers on two lower-tier GPUs and sends embeddings over NIXL to PD workers on four other GPUs.

Launching

cd $DYNAMO_HOME/examples/backends/vllm
# E/PD
bash launch/disagg_multimodal_e_pd.sh --model "Qwen/Qwen3-VL-30B-A3B-Instruct-FP8"
# E/P/D
bash launch/disagg_multimodal_epd.sh --model "Qwen/Qwen3-VL-30B-A3B-Instruct-FP8"

See the backend-specific documentation (vLLM, TRT-LLM, SGLang) for full configuration details and component flags.

Support Matrix

BackendE/PDE/P/DNotes
vLLMYesYesSeparate encode worker currently handles image_url inputs; video_url inputs stay on the prefill/PD path
TensorRT-LLMYesYesSupports image URLs (via MultimodalEncoder) and pre-computed embeddings (via NIXL)
SGLangYesYesNIXL for embeddings; bootstrap mechanism for P/D KV transfer