Dynamo v1.4.0

Dynamo v1.4.0 is the 17th feature release of the open-source distributed inference platform, spanning 640 merged PRs from 127 contributors. It takes Dynamo Router multi-datacenter with a sequenced KV relay, peer reservation replay, and endpoint-scoped event transport. It opens an experimental vLLM-compatible /inference/v1/generate token-in/token-out endpoint on the Dynamo Frontend alongside a default-on tokenizer prefix cache. vLLM-Omni pipelines gain NIXL RDMA disaggregation for true multi-node autoregressive-to-diffusion serving, the Spica discrete-event simulation framework joins the repository, and the engine fleet moves to vLLM v0.26.0, SGLang v0.5.16, TensorRT-LLM 1.3.0rc22, and NIXL v1.3.x. The release also ships a rebuilt documentation website with tab-based navigation and native Simplified Chinese localization.

Get v1.4.0

Pull, deploy, and install with every artifact pinned to the v1.4.0 release set.

docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.4.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.4.0.tgz
uv pip install "ai-dynamo[sglang]==1.4.0"

docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.4.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.4.0.tgz
# TensorRT-LLM ships via the NGC container

docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.4.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.4.0.tgz
uv pip install "ai-dynamo[vllm]==1.4.0"

Highlights

Features & Improvements

Frontend

Generate Endpoint

• Experimental /inference/v1/generate Endpoint: Added a vLLM-compatible /inference/v1/generate token-in/token-out endpoint on the Dynamo frontend (#11108), then dispatched real requests through it with Generate capability gating, tokenizer-free frontend registration, and sanitized cancellations and errors (#11167). The endpoint remains experimental and unary-only.

Tokenizer & Chat Templates

• Tokenizer Prefix Cache Enablement: Enabled the process-local L1 tokenizer prefix cache by default unless DYN_TOKENIZER_CACHE is exactly 0, raising the default byte budget from 50 MiB to 64 MiB (#11078). Extended the cache to supported TikToken models by forwarding exact registered special-token boundaries (#11499).
• Chat-Template Thinking Defaults: Added deployment-level thinking defaults for chat templates (#12440), and honored reasoning_effort for templates that gate reasoning through enable_thinking by deriving enable_thinking=false for reasoning_effort="none" and true for other supplied values, with explicit chat_template_args.enable_thinking values taking precedence and requests without reasoning_effort left unchanged (#11691).
• Dynamo Protocols 3.1.0 Upgrade: Updated Dynamo to dynamo-protocols 3.1.0 with compatible parser, renderer, and tokenizer versions, propagated tokenizer-cache initialization failures instead of hiding them, and adapted Anthropic, OpenAI Responses, and media preprocessing to protocol 3.x optional media URL fields (#11978).

OpenAI API & Validation

• Configurable Overload Rejection Status: Added DYN_HTTP_OVERLOAD_STATUS_CODE so operators can choose the HTTP status returned for Frontend overload and admission-control rejections, keeping 529 as the default and accepting 503 (#11249). Rejected informational status codes in the parser, falling back to 529 instead of surfacing HTTP 500 (#12738).

Request Plane & Extensions

• Python Frontend Route Extensions: Added a dynamo.llm.FrontendRoute hook so external packages can register GET routes on the Frontend via --frontend-route-extension or DYN_FRONTEND_ROUTE_EXTENSIONS, without a custom binary. Routes run on a bounded pool and shed load with 503s, keeping handlers off the hot path (#11626).

General

• Squeeze-Evolve Multi-Model Service: Added an experimental dynamo.squeeze_evolve component that serves verifier-free evolutionary test-time scaling as a chat model, routing candidate groups across model tiers via per-tier KvRouter instances. It is opt-in and not wired into any default frontend path (#10785).

Multimodal & Diffusion

• vLLM-Omni Realtime Worker: Added a realtime bidirectional vLLM-Omni worker exposing a WebSocket /v1/realtime endpoint that streams transcript and output audio deltas (#10166), and fixed an AttributeError on newer vLLM-Omni versions when reading multimodal_output (#12667).
• Unified vLLM Multimodal Pipeline: Added image, video, and audio handling to the aggregated unified vLLM backend (#11267) with modality-aware multimodal routing metadata (#10375). Enabled the vLLM multimodal CPU embedding cache (#11268), added frontend media discovery plus SHM and NIXL transfer fallback (#11269), and brought multimodal prefill and decode parity to the unified backend (#11270).
• Pluggable Custom Vision Encoder: Added a VisionEncoderBackend ABC, the build_mixed_embeds assembler, and a --custom-encoder-class flag (#10910), then wired the contract into serving so an aggregated dynamo.vllm worker runs a user-supplied encoder in-process without a separate encode worker (#10832). Added adapters that build the final vLLM EmbedsPrompt or TokensPrompt directly (#11847).
• Cached Multimodal UUID Passthrough: Added UuidOnly and multi_modal_uuids support, preserving user-provided image UUIDs and image-slot alignment through the vLLM worker (#11943), rejected non-null multimodal UUIDs on the SGLang and TensorRT-LLM paths (#11944), and parsed canonical and deprecated image UUID forms in the Rust and Python frontends (#11958).
• Media Decoder Installer And Diagnostics: Added an operator-run installer for the media-decoder packages the runtime images omit, invoked as python -m dynamo.common.utils.install_media_decoders <vllm|sglang|trtllm> with --packages, --pip-args, --timeout-s, and --dry-run flags (#12051). Reworked decode errors to name the missing package, installer command, and hardware alternative (#12725).
• NVDEC Hardware Video Decode: H.264/H.265 video inputs now decode on the GPU via NVDEC through PyNvVideoCodec, keeping decode work off the SMs beyond YUV-to-RGB conversion. DYN_DISABLE_NVDEC opts out and DYN_NVDEC_GPU_ID selects the decode GPU (#11836).
• Encode-Worker Request Routing: Added a backend-agnostic encoder routing hop that sends multimodal requests to a surface-less Encode worker before aggregated or prefill execution, with an optional round-robin EncoderRouter (#11460). Forced the vLLM V1 model runner in the disaggregated EPD encode worker to work around a vLLM 0.25.1 startup crash (#11913).
• Video Embedding Cache Benchmark: Added a video-understanding benchmark workload for SGLang multimodal embedding cache reuse, including video-single-turn and video-sliding-window JSONL generation and an E/PD sweep config comparing cache-off vs cache-on, letting users evaluate SGLang video cache performance (#10000).
• NIXL Connector for vLLM-Omni: Added a NIXL RDMA-based tensor transport connector for vLLM-Omni, enabling autoregressive-to-diffusion (AR->DIT) pipelines such as GLM-Image to run disaggregated across multiple nodes and device types instead of requiring shared memory on a single machine (#10071).
• Cross-Request Vision Batching: Added a ThreadedMicroBatcher and batcher-backed AsyncVisionEncoder that combine image forwards from concurrent requests into thread-affine, cost-bounded batches, improving GPU utilization for the custom encoder path. Canceled requests are tombstoned before dispatch so abandoned input cannot affect in-flight requests sharing a later batch (#11037).
• TensorRT-LLM Video URL Support: Added video_url multimodal input handling to the TensorRT-LLM backend, decoding video URLs into VideoData instead of silently dropping them, mirroring the image_url path’s validation and controlling frame count via DYN_MM_VIDEO_NUM_FRAMES (#11896).

Scheduling

Multi-DC Indexing & Relay

• Radix Tree Memory Reduction: Replaced each positional radix bucket’s inner DashMap with FxHashMap, since the outer DashMap already served as the concurrency boundary and inner sharding added no benefit. This reduced peak Mocker RSS by 5.51 GiB (35.3%) in a 32-worker A/B test (#12161).
• CRTC Compressed Radix Tree Indexer: Replaced each CRTC node’s eager DashMap with adaptive child storage, improving full-trace throughput from 9.04M to 13.49M block ops/s (#11459). Added a router-local Relay boundary for exact per-datacenter aggregation (#11435), renamed tenant_id to routing_group across APIs (#11471), closed stale-shape races (#11169), and skipped shape locks for already-internal nodes (#11102).
• Arena-Backed Block Tracker Rework: Replaced per-block Arc ownership in the KV Router block tracker with a SlotMap-arena block chain (#11508), removed the PromptRegistry mutation lookup (#11515), and moved to compressed radix-tree edges (#11644). Streamlined BlockTracker::release (#11568), fixed a leak in ConcurrentRadixTreeCompressed (#11785), and preserved worker and DP-rank coverage on cold restores (#11190).
• Missing KV Event Publisher Diagnostics: Added the optional kv_event_publishing_enabled runtime capability across backends with a warning-only source-health monitor and stable structured codes (#12095), plus a Router diagnostic that logs an actionable error when expected worker or DP query endpoints never appear on the local-indexer recovery path (#11194).
• Endpoint-Scoped Relay Event Transport: Added endpoint-scoped EventPublisher/EventSubscriber constructors that thread namespace/component/endpoint identity through discovery and transport (#11841), added a domain-scoped KV Relay with a lane-sticky global indexer for multi-DC prefix routing (#11793), and fixed a startup TypeError in the relay’s health endpoint (#12948).
• Planner Predictor Data Inputs: Added a KV-cache-read dimension to the vLLM prefill self-benchmark grid, gated behind --benchmark-prefill-kv-read-granularity and DYN_BENCHMARK_PREFILL_KV_READ_GRANULARITY (#11150). Added native dynamo.request.trace.v1 support to Planner load-predictor warmup with schema auto-detection and rejection of malformed traces (#11842).
• Pool-Scoped KV DC Relay: Added a pool-scoped KV DC Relay runtime mapping each serving endpoint to one Dynamo PoolId, with a PoolRegistry owning per-pool Relay actors and CKF state. This keeps independent endpoints serving the same model isolated and recovering from stream failures (#12199).

Selection & Reservation Replay

• Active Sequence Replica Event Pipeline: Batched frontend active-sequence replica-sync lifecycle events on ZMQ with the shared 256-event and 1 ms policy (#11920), added a lifecycle-owned direct-ZMQ fan-in preserving per-source FIFO (#11967), and replaced the async publication API with a bounded synchronous enqueue that drops the newest event on overflow (#11992).
• Standalone EPP Selector Mode: Added a standalone EPP mode selected by DYN_EPP_MODE with EppStandaloneConfig validation (#11070), embedded the dynamo-kv-router SelectionService with optional multi-EPP replication via DYN_EPP_PEER_SERVICE (#11541), wired an in-process EppRouter without DistributedRuntime, etcd, or NATS (#11542), and preserved token-id completion prompts in the EPP scorer request (#10872).
• Selection Booking Replay By Id: Added a SelectionCache keyed by (model, routing_group, selection_id) so a follow-up POST /reservations replays the cached POST /select booking instead of re-sending and re-hashing a long prompt (#11416), with select_and_reserve returning the normalized booking fields so peer ingress replicas can replicate it (#12335).
• Session Affinity Across Replicas: Propagated agent_context.session_id through live router selection and Mocker replay into SchedulingRequest (#11089), and synchronized session-affinity bindings automatically whenever the affinity TTL is configured, with first-live-binding-wins semantics (#11750). Moved replica tasks under the affinity coordinator and upgraded the end-to-end test to an adversarial two-frontend scenario (#11750).
• Decode Affinity Scoring Path: Added decode affinity scoring to the Router’s worker selection (#11720), with an experimental block-equivalent decode cost term plumbed through config, CLI, and bindings (#12158). Stamped request_id and worker_type on cost-breakdown log rows (#12370) and extended the same stamping to the decode-affinity branch (#12955).
• Provider-Keyed Tracking Hashes: Added an opt-in keyed-xxh3-v1 identity mode for router tracking, backed by a zeroized 32-byte provider key and managed key ID, preserving hashes and wire formats. Standalone selection honors router_assume_kv_reuse=false with random tracking identities; rotation requires coordinated key changes and restarts (#11548).
• ISL-Based Aggregate Routing: Added an optional input sequence length (ISL) dimension to aggregate GlobalRouter pool selection, allowing requests to route across pools based on token count in addition to existing TTFT x ITL configurations. Existing 2D TTFT x ITL routing remains unchanged when ISL settings are not configured (#11835).

KV Events & Source Health

• Concurrent KV Event Ingestion: Delivered direct-ZMQ KV events through one FIFO task per publisher, bypassing the lossy broadcast hop (#11938). Preserved native SGLang and vLLM event-list boundaries (#11466), published ordered RouterEvent vectors in partitioned batches (#11776), batched deferred Mocker visibility boundaries (#11860), and restored KV-aware routing for multinode SGLang workers on Kubernetes (#12185).
• Cache-Salt Multi-Tenant KV Isolation: Added cache-salt-aware KV routing so requests carrying a tenant namespace (via x-tenant-id, nvext.cache_salt, or a compatibility cache_salt field) hash and route independently across the Router, vLLM, and TensorRT-LLM KV events, preventing cross-tenant cache reuse while preserving existing unsalted behavior (#8197).
• Forward-Pass Metrics Tracing: Added opt-in, best-effort persistence of forward-pass metrics to bounded rotating gzip JSONL traces for vLLM, SGLang, TensorRT-LLM, and Mocker publishers, exposed via --fpm-trace/--no-fpm-trace and DYN_FPM_TRACE, giving operators durable forward-pass records without adding disk I/O to the inference path (#11110).

Routing Policies

• Router Queue Admission Strategy: Added exact-worker lanes to the policy-class queue (#11363) with deferred blocked-lane rechecks (#11484), a public PolicyClassAdmissionStrategy contract (#11434), and a live RequestProgress reader (#11615). Kept Router queueing off unless a threshold is configured (#11599) and replaced --admission-control with opt-in thresholds, ignoring the legacy flag and DYN_ADMISSION_CONTROL with a warning (#11276).
• Conditional Disaggregation Routing Policy: Added a conditional disaggregation policy kernel to the Router (#11718) with backend handling for the conditional disagg bypass annotation (#11719). Deprecated --enforce-disagg and DYN_ENFORCE_DISAGG as accepted no-ops (#11160), emitting the deprecation warning immediately for both forms (#11800).
• Configurable Active Request Expiry: Added the DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS environment variable to configure the KV-router’s stale active-request cleanup guard, preventing long-context agentic requests from having their live record removed before completion when they exceed the previous hard-coded 300-second limit (#11303).
• Native KV Offloading Capacity: Added a backend-neutral native_offloading_capacity.total_tokens metadata contract for Rust and Python routing consumers, accepting SGLang capacity only from the scheduler’s realized host pool and translating it by write policy to avoid double counting GPU capacity in router budgets (#11321).
• Amplified Overlap Score Credit: Allowed the KV Router’s --router-kv-overlap-score-credit value to exceed 1.0 and removed the zero floor on adjusted prefill blocks, so amplified overlap credit can offset decode load and improve cache-hit routing without adding new configuration surface (#11714).
• Native Multimodal KV Routing: Moved TensorRT-LLM aggregated multimodal KV routing onto Dynamo’s Rust Frontend and KV Router, removing the separate Python MM-router sidecar and updating worker integration for TensorRT-LLM v1.3.0rc22. This simplifies deployment to a single native request path while preserving KV cache reuse across repeated image requests (#11737).

General

• HiCache Host Capacity Budget: Updated ThunderAgent to derive its retention budget from GPU KV plus native HiCache host capacity instead of device KV alone, aligning pause/resume behavior with the memory SGLang can actually retain (#11185).

Performance Modeling & Replay

Spica Simulation

• Experimental Spica Simulator Integration: Moved the experimental Spica simulator from aiconfigurator into Dynamo as an independent aisimulate distribution with import aisimulate.spica and CLI python -m aisimulate.spica (#11923), and stopped emitting synthetic arrival intervals for closed-loop workloads so Replay receives only replay_concurrency (#12824).
• SATF Request-Trace Conversion: Added a Rust API and CLI (request_trace_to_satf) that converts Dynamo request-trace JSONL and gzip JSONL shards into SATF 2.0 performance-layer artifacts, preserving token counts, timing, KV block identity, request metrics, and tool-call structure for downstream simulation and analysis without altering the existing capture format (#11375).

Mocker

• Native G1 Block Manager: Added a selectable native G1 KV manager for the shared vLLM and TensorRT-LLM mock schedulers (#12033), defaulted both backends to native G1 with fallback to KVBM when G2/G3/G4 offload is required (#12222), flattened native G1 request token representation (#12248), and fixed a build failure via resolved_g1_backend() (#12336).
• SGLang Mocker Scheduling Fidelity: Streamlined SGLang radix scheduling in the Mocker by keeping each request in one append-only u32 token sequence so decode caching borrows the retained prefix (#12043), and modeled SGLang KV allocation by page instead of individual token-slot IDs (#12108).
• Mocker KV Bookkeeping Overhead: Reduced Mocker KV-management overhead in high-concurrency disaggregated workloads by batching terminal dereference signals and using canonical immutable block handles (#11095). Replaced the native VllmBlockPool inactive-copy BTreeMap with an intrusive linked LRU for constant-time operations, with a balanced AgentX A/B measuring a 22.2% median output-token throughput improvement (#12244).
• Mocker Max Model Length: Added explicit max_model_len support to the vLLM Mocker and replay so they match the frontend, Router, and real worker sequence-length behavior. Requests exceeding the context limit are rejected up front, while mid-generation hits complete with finish_reason="length" (#11069).

Trace Replay & Benchmarks

• Poisson Synthetic Replay Arrivals: Added Poisson synthetic replay driven by request rate, alongside fixed-interval and concurrency modes; an independent arrival seed means switching modes changes only timestamps. Replay now requires one of request rate, interval, or concurrency, removing the implicit 1 ms fallback (#12062).

Kubernetes Deployment

Inference Gateway (EPP)

• Standalone EPP Worker Discovery: Added discovery of standalone EPP workers from the gateway-targeted GAIE InferencePool via a kube-rs reflector index of Ready, selector-matching workers (#11072), with a topology adapter that converts each ready worker into a WorkerRegistration and reconciles the worker set with the in-process Selector (#11074).

Operator & CRDs

• Grove Update Strategy Controls: Added spec.grove.updateStrategy.type to DynamoGraphDeployment with RollingRecreate and OnDelete values, surfacing OnDelete rollout progress through a DGD condition and events (#11085), and added component-scoped Recreate support via the nvidia.com/deployment-strategy pod template annotation (#11909).
• CRD apiVersion Converter Utility: Added deploy/utils/convert_api_version.py, a CLI that converts nvidia.com/v1alpha1 manifests to nvidia.com/v1beta1 by POSTing a ConversionReview to the Operator’s existing conversion webhook (#10657). Fixed the power-agent Helm chart, where a whitespace trim broke the default helm install (#12651).
• Grove Volcano Scheduler Support: Added Operator-controlled Volcano scheduler integration for Grove-backed DynamoGraphDeployment workflows, setting schedulerName: volcano on Grove clique pods and mapping the nvidia.com/volcano-queue annotation to Grove’s PodCliqueSet metadata, with startup validation preventing simultaneous KAI-Scheduler and Volcano configuration (#8225).
• DynamoGraphDeployment Placement Score Status: Added status.placementScore and status.placementScoreState fields to the DynamoGraphDeployment API, mirrored across v1alpha1 and v1beta1 with conversion wiring, and reported non-failing placement states during reconcile on the Grove pathway. Full score reporting awaits Grove exposing a consumable score field. (#11057).
• Fine-Grained DGD Readiness Reasons: Added classification of the DynamoGraphDeployment Ready condition into specific reasons (insufficient_capacity, pods_not_ready, updating, mixed_not_ready_reasons) derived from Grove status fields, plus an optional scheduledReplicas field on component status, so operators can diagnose why a Grove-backed deployment is stuck without inspecting Grove resources directly (#11131).
• Operator RuntimeVersionOverride Backport: Added runtimeVersionOverride support for DGD (DynamoGraphDeployment) components and DGDR-generated deployments, backported to release/1.4.0, letting users pin a specific runtime version for these components (#12427).

GPU Memory Service & Snapshot

• GMS Shadow Failover Hardening: Added dynamo_component_engine_failover_* counters and state gauges for intra-pod shadow-engine failover with a Grafana dashboard (#11664), aliased all scratch KV allocations to a single granule of device memory (#11911), and surfaced scratch-KV engagement in shadow worker logs (#11724).
• VMM Device Abstraction Layer: Introduced a device-agnostic Virtual Memory Management (VMM) abstraction in the GPU Memory Service with a --device-kind CLI option and VMMDevice protocol, preserving existing CUDA behavior while laying groundwork for XPU support, which raises NotImplementedError until Phase 2 (#9788).

General

• XPU DRA Deployment Examples: Added XPU Dynamic Resource Allocation deployment YAMLs for aggregated and disaggregated vLLM serving, including tracing and Planner variants, giving users ready-made configs for aggregated serving, OpenTelemetry tracing, NIXL KV transfer, and Planner-based disaggregation on XPU. (#9253).

Fault Tolerance & Observability

• OTLP Log Export Sink: Added an opt-in otel audit sink that exports chat-completion audit records as OTLP LogRecords, enabled via DYN_AUDIT_SINKS (#9390), and decoupled OTEL_EXPORT_ENABLED from DYN_LOGGING_JSONL so OTLP export initializes regardless of the local log format (#10817).
• Unified Request-Trace Payload Capture: Folded audit-style payload logging into the request trace pipeline, emitting a single dynamo.request.trace.v1 stream with payload rows selected via DYN_REQUEST_TRACE_RECORDS (#11180), and added allowlisted HTTP request-header capture through DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST (#11386).
• Frontend Multimodal Input Metrics: Added per-request counts of image_url, video_url, and audio_url content parts as frontend Prometheus histograms and request-completed log fields (#11166), followed by the dynamo_frontend_image_tokens_per_request histogram of calculated image-placeholder token counts (#12065).
• Local Resource Monitor Caching: Cached process selection and aggregation in the local resource monitor so dashboard deltas no longer rescan every retained process series, and disconnected hidden dashboard tabs from Socket.IO until shown again, reducing background traffic and CPU use (#11336).
• Tokenizer Cache Token Metrics: Added per-model dynamo_frontend_tokenizer_cache_cached_tokens_total and dynamo_frontend_tokenizer_cache_uncached_tokens_total counters alongside the existing aggregate hit and miss metrics, letting operators measure token-level cache reuse per served model instead of inferring it from aggregate counts (#11353).
• Buffered ITL Metrics Collection: Reduced inter-token latency metrics overhead by buffering observations in a request-local Prometheus histogram, flushing every 64 samples instead of synchronizing on every output token, cutting the ITL histogram on-CPU cost by 71.9% while preserving the existing Prometheus schema (#11569).
• Dynamic Log Filter Warning: Added a stderr warning emitted when a dynamic span or field selector forces logging to fall back from the lock-free Targets filter to EnvFilter, explaining the streaming-performance risk and how to restore the fast path (#11974).
• Configurable Worker Inhibition Duration: Added DYN_RUNTIME_INHIBITED_DURATION_SECS to configure the local worker inhibition window, keeping the five-second default when unset and allowing 0 to disable local inhibition(#11993).

Engines

vLLM

• Graph-Aware Self-Benchmark Sampling: Replaced linear per-request sweeps in the vLLM self-benchmark with deterministic prefill and decode sampling derived from the resolved PIECEWISE/FULL CUDA Graph configuration (#11509). Added an option to override the benchmark grid (#11713) and filtered infeasible EAGLE prefill KV-read coordinates (#12836).
• Unified Backend Elastic EP: Added headless multi-node worker support and live elastic expert-parallel (EP) scaling via scale_elastic_ep to the vLLM unified backend, bringing it to parity with the legacy path and letting deployments scale data-parallel size up or down without restarting service (#10834).
• vLLM Multiple Served Names: Enabled the vLLM backend to accept multiple --served-model-name values, registering the first as primary and the rest as aliases, matching SGLang. A single worker now serves several model names, with /v1/models and chat completions resolving aliases to the primary (#11611).
• vLLM Render Client Support: Added a pooled HTTP client in the standalone External Processor (EPP) that calls vLLM’s /v1/chat/completions/render endpoint for tokenization, configurable via DYN_EPP_TOKENIZER_SERVICE_URL and DYN_EPP_TOKENIZATION_TIMEOUT_MS, replacing an earlier offline tokenizer approach and classifying failures as typed Unavailable, Timeout, UpstreamStatus, or InvalidResponse errors (#11827).

SGLang

• Multiple Served Model Names: Added support for registering an SGLang worker under multiple served model names via --served-model-name, canonicalizing alias requests to the primary name so engine routing, Prometheus metrics, and the OpenAI response.model field stay consistent (#11141).

TensorRT-LLM

• Engine-Owned Conversation-Aware ADP Routing: Added opt-in TensorRT-LLM engine-owned conversation-affinity attention-DP routing, deriving the conversation id from agent_context.session_id and passing it as ConversationParams (#11609), with a --conversation-affinity flag and DYN_ENGINE_CONV_AFFINITY environment variable forcing engine-side assignment (#11705).
• Prefill Handoff And Event Batching: Added generic Rust and Python KvEventPublisher.publish_batch APIs so one ordered source batch reaches the KV event processor as a single input (#11783), and skipped duplicate prompt metadata in the legacy TensorRT-LLM prefill handoff for text-only requests (#11202).

Infrastructure Modernization

• Dependency and Service Refresh: Aligned aiohttp to 3.14.3 or later across all container images, including the TensorRT-LLM runtime’s system-site install (#13132, #13195, #13227, #13256), bumped etcd to v3.5.33 and NATS Server to v2.12.14 (#13016, #13225), moved vLLM images to NIXL v1.3.2 (#12882, #13185), and adopted the published AIConfigurator v0.11.0 release (#13280).
• TCP Request-Plane Fast Path: Removed two large request-payload copies from the TCP request-plane frontend send path for payloads of at least 4096 bytes via vectored socket writes (#10519), and replaced the Tokio mutex guarding the TCP stream registry with parking_lot::Mutex (#11065).
• Python Request-Plane Codec Transcoding: Added a statically dispatched ingress payload adapter that transcodes request-plane wire bytes directly to and from Python objects, making MessagePack the default with JSON as an explicit fallback (#11104), interned the six annotated-response envelope keys (#11331), and added per-worker codec negotiation via discovery metadata (#12658).
• Client TCP Address Lookup: Added Client.instance_tcp_addresses() to the Python bindings, returning a dict mapping instance id to TCP transport address for an endpoint’s registered instances, letting callers reach peer worker nodes without reading the discovery/etcd registry layout directly (#11540).

Hardware

• XPU Deployment Sanity Checks: Extended dev/sanity_check.py to detect and report Intel XPU systems alongside NVIDIA systems, and added xpu-smi to XPU Docker images, giving users GPU, framework, and filesystem diagnostics for Intel XPU deployments (#7702).
• vLLM XPU Support Enablement: Resolved the accelerator device from vLLM’s current platform in the multimodal embedding cache connector so CPU-cached embedding loads work on both CUDA and XPU (#9938), and updated the XPU aggregated examples for vLLM XPU 0.24.0 by removing the --block-size override (#11229).

KV Block Manager

• Unified KV Block Clearing: Added a POST /engine/control/clear_kv_blocks endpoint for unified vLLM workers that resets the prefix cache and connector state without preempting active requests. Supported aggregated, prefill, and decode workers, and propagated cached-token usage so clients can verify cache hits and post-clear misses (#11451).

General

• vLLM v0.26.0 Upgrade: Upgraded the pinned vLLM to v0.26.0 with refreshed CUDA and CPU runtime images, FlashInfer 0.6.14, and NIXL v1.3.1, later moved to v1.3.2 (#12202, #11606), and moved the XPU container to the vllm/vllm-openai-xpu v0.26.0 image (#12190). The upgrade brings reasoning-parser configuration and structured-output handling to the Rust vLLM backend.
• TensorRT-LLM v1.3.0rc22 Upgrade: Upgraded TensorRT-LLM to v1.3.0rc22 across supported runtime configurations and platforms with updated baseline SBOM records (#11086, #11763, #12071), including improved multimodal request handling and image-token range detection for TensorRT-LLM KV event processing.
• SGLang v0.5.16 Upgrade: Upgraded the SGLang backend and CUDA 13 runtime to v0.5.16 on the lmsysorg/sglang:v0.5.16-cu130-runtime image with NIXL v1.3.0 pinned for disaggregated serving (#11596, #12081), setting incremental_streaming_output before SGLang resolves its immutable startup record and removing the expired 0.5.14 profiling compatibility path.
• dynamo-parsers 7.0.1 Upgrade: Upgraded dynamo-parsers to 7.0.1 in the workspace and Python bindings (#12147, #12451, #12509), aligning the frontend crate graph on dynamo-protocols 4.0.0 and dynamo-renderer 3.0.0 and picking up the DeepSeek fix that stops tool-call markup from leaking into reasoning content.
• Inline Compliance Artifact Coverage: Extended the inline license/SBOM compliance pipeline to the Planner and Operator images (#11328) and the frontend and snapshot images (#11533). Added per-build OSRB CSV diffs against context-dependent baselines (#11240) and passed the IRSA sccache secrets to compliance-extract rebuilds (#11181).
• Power Agent DCGM Actuator: Added an opt-in DCGM actuator for the Power Agent, letting operators set agent.actuator=dcgm to write power caps through the cluster nvidia-dcgm hostengine, with NVML remaining the default and cap writes anchored to a transaction-verified GPU UUID (#9790).
• Self-Hosted MDC Default: Flipped DYN_SELF_HOST_METADATA to on by default so workers advertise their model card (MDC) files over the system_status_server instead of requiring shared storage, and added a startup hard-fail when self-hosting is enabled but DYN_SYSTEM_PORT is unset instead of silently degrading hf:// MDC advertisement (#11417).
• Hugging Face Hub LoRA Source: Added native hf://<repo>[@revision] support for dynamic LoRA loading, downloading adapters into the standard Hugging Face Hub cache with revision pinning, timeout limits, and integrity checks instead of requiring a separate copy under DYN_LORA_PATH (#11431).
• Instance Transport Object Model: Added Client.instances() to the Python bindings, returning read-only Instance snapshot objects with instance_id, namespace, component, endpoint, device_type, and transport attributes, preserving all transport types including NATS instead of flattening addresses to opaque TCP-only strings (#11617).
• Static Logging Filter Fast Path: Switched to a metadata-only Targets filter for static DYN_LOG and configured log filter directives, falling back to EnvFilter only for dynamic span or field directives, removing a shared span-matcher lock from the request-span lifecycle and reducing frontend CPU overhead on streaming responses (#11820).
• EFA Installer Version Bump: Updated the AWS EFA Installer to version 1.49.0 and reverted the prior libfabric override so the AWS container now uses the installer’s provided libfabric, improving compatibility with EFA networking (#12010).

Recipes

• Model-Agnostic Accuracy Check: Added recipes/accuracy/, an AIPerf-based Kubernetes Job that runs a public benchmark against any deployed Dynamo recipe and grades the answers, with a methodology README (#12495, #13215).
• GPT-OSS-120B vLLM Deployment Recipes: Added GPT-OSS-120B recipes for aggregated and disaggregated vLLM serving on B200 and H200 GPUs, covering speculative decoding, KV-cache routing and offload, reasoning, tool calling, automated model downloads, and an AIPerf trace-replay benchmarking workflow with documented prerequisites, quick start, performance results, and known limitations (#11506). Fixed the disaggregated deployments so they no longer force etcd discovery at a hardcoded endpoint, letting the cluster discovery backend injected by the Operator apply, while keeping the NATS message bus setting (#12070).
• Hugging Face Hub Pin Bump: Upgraded the huggingface_hub CLI pin from 1.11.0 to 1.16.4 in model-cache and vLLM LoRA helper job YAML files, fixing a missing click dependency that broke the hf download command (#10986).
• ARM Recipe Build Tools: Added build-essential installation to ARM/GB200-facing benchmark recipes so aiperf>=0.8.0 can build its crick dependency from source, since crick lacks a Linux aarch64 wheel (#11278).
• DSV4 Agentic vLLM Recipes: Added DeepSeek-V4-Flash and DeepSeek-V4-Pro deployment recipes combining Dynamo with vLLM, covering aggregated and P/D-disaggregated profiles on B200 and H200 for agentic workloads, plus model-cache manifests and an AIPerf Moontrace perf job. Disaggregated variants move KV over NIXL GPU-Direct RDMA (#11429).
• TokenSpeed FlashInfer Cache Fix: Pinned the TokenSpeed recipe’s base image by immutable digest, fixed the source revision, and aligned the FlashInfer JIT cache version required before building kernels, preventing version mismatches in kimi-k2.5 builds (#11465).
• Inkling NVFP4 B200 Recipe: Added a recipe for thinkingmachines/Inkling-NVFP4 with Dynamo plus SGLang aggregated serving on one 8x B200 node, including NVFP4 quantization, FA4 attention, EAGLE speculative decoding, and a model-cache download job. It documents a required DYN_FORWARDPASS_METRIC_PORT override avoiding a scheduler crash (#11743).
• GLM-5.2 SGLang Recipes: Added GLM-5.2 SGLang aggregated and disaggregated serving recipes for B200 and H200 clusters, including model-cache setup, EAGLE speculative decoding, HiCache CPU offload, NVFP4/FP8 quantization, and an AIPerf trace-replay benchmark workflow, along with fixes for reasoning-parser configuration, a non-stream logprobs 500 error, and trace-staging symlink handling. (#11926).
• Kimi-K3 Recipe Publication: Brought the Kimi-K3 recipe assets (README, model-cache config, and vLLM deploy manifests for GB200/GB300 aggregated and disaggregated setups) onto main and added its row to the Available Recipes table, so users on main can now discover and deploy Kimi-K3 without switching to the release branch (#12228).

Bug Fixes

Frontend

• LoRA Unload Resource Release: Fixed the Frontend to release the worker connection and associated memory after each LoRA unload, eliminating unbounded RSS growth of roughly 95 MB per load/unload cycle (#12743, #13203).
• FinishReason Error Propagation: Fixed the Frontend to read the string form of FinishReason that workers emit, so request-level error messages reach the caller instead of a bare HTTP 500 (#13118, #13212).
• MDC Version Compatibility Fix: Preserved compatibility for Model Discovery Card (MDC) versions across rolling upgrades by accepting legacy v1.2 context_length fallbacks and omitting default-valued post-v1.3 router fields so v1.3 frontends can deserialize v1.4 MDCs, while keeping removed v1.3 router behavior like durable_kv_events=true failing closed (#12801).
• Hidden Disabled Batch Routes: Fixed the OpenAPI specification to omit Batch and Files API routes when their internal enable flag is off, so disabled surfaces no longer advertise themselves (#12760, #13122).
• Chat Template Kwargs Plumbing: Fixed the vLLM chat processor so per-request chat_template_kwargs reach the chat template (#11729), threaded the server-wide --default-chat-template-kwargs flag through with per-request values overriding defaults (#11731), and rejected a chat_template nested inside kwargs, closing a validation bypass (#11755).
• Streaming Tool-Call Emission Fixes: Reworked the streaming converter for the Anthropic Messages and OpenAI Responses formats to accumulate fragmented tool-call identities and argument deltas, publishing each complete block on finish (#10988), and synthesized a finish_reason: "tool_calls" chunk when the upstream stream omitted its terminal reason (#11045).
• Guided JSON Reasoning Parsing: Fixed guided decoding for force-reasoning parsers, preserving both reasoning_content and structured tool calls for tool_choice=required and named tool calls (#11205), and routed response_format: json_schema output to content rather than reasoning_content with SGLang reasoning enabled (#11512).
• Token Usage Accounting Fixes: Corrected Anthropic /v1/messages input_tokens accounting so cached tokens are no longer double-counted (#11030), propagated completion_tokens_details including reasoning_tokens into Chat Completions responses (#11027), and added DYN_ENABLE_FORCE_INCLUDE_USAGE to require usage in streaming responses even when clients omit include_usage (#12017).
• Frontend Request Validation Hardening: Mapped InvalidArgument errors from the annotated first-event path to HTTP 400 (#11453) and converted preserved aggregation errors on /v1/audio/speech so worker rejections return 4xx (#12832). Restored boundary validation so max_tokens: 0 returns HTTP 400 (#11394), and tolerated raw control bytes in JSON string fields on the OpenAI endpoints (#11114).
• Responses API Event Semantics: Mapped length-limited generation to incomplete response states, emitting response.incomplete with max_output_tokens as the reason (#12182), emitted requested reasoning summaries through the full ordered streaming lifecycle (#12183), and rejected completion_token_ids and prompt_logprobs requested via nvext.extra_fields with a field-specific 501 (#11253).
• Release 1.4.0 Frontend Backports: Backported validation of converted OpenAI Responses and Anthropic requests, returning 400 for malformed inputs and 501 for unsupported features (#12904), and rejected non-progressing guided JSON cycles (#12853). Forwarded vLLM tool-call and structured-output constraints to workers (#12877), preserved explicit top_logprobs=0 (#12681), and converted SGLang log_probs and top_logprobs to the OpenAI contract (#12988).
• Model Registry Reconciliation: Added periodic reconciliation that queries all discovered models every 30 seconds and retries incomplete registrations, so a frontend’s local model registry recovers from missed discovery events or transient registration failures without manual intervention (#11092).
• Async Tokenization Offload: Moved prompt tokenization off the Frontend’s tokio event loop onto the blocking thread pool, matching the embedding-path offload. This prevents BPE encoding of long prompts from stalling the event loop and wrongly marking healthy workers as down in disaggregated serving (#11200).
• MiniMax M2 Tool-Calling Fix: Fixed forced and named tool calling for MiniMax M2 with reasoning by wiring minimax_m2 into Dynamo’s force-reasoning and guided-output parsing paths, so tool output is separated from reasoning instead of misparsed. Docs now recommend minimax_m2 over deprecated minimax_append_think (#11554).
• Kimi Multi-Turn Reasoning Fix: Fixed post-tool reasoning parsing for Kimi K2.5/2.6 so private reasoning and the </think> marker no longer leak into content, and normalized thinking/enable_thinking values consistently across prompt rendering, the SGLang reasoning gate, and the postprocessor (#11653).
• SGLang Reasoning Delimiter Fix: Fixed the custom SGLang chat processor to preserve special delimiter tokens during incremental detokenization when a tool-call or reasoning parser is active, so reasoning content populates correctly in streaming output (#11748).
• Startup File-Descriptor Limit: Raised the process’s soft RLIMIT_NOFILE at frontend startup, bounded by the environment’s hard limit, to prevent the TCP accept() loop from spiraling into EMFILE errors and flooding logs under high connection concurrency (#11802).
• Tool-Result Media Preservation: Fixed Anthropic tool results so embedded images are no longer reduced to text-only content, and added support for image, video, and audio URL parts in OpenAI-compatible tool messages, routing them through the existing user-media preprocessing path (#12180).
• Parallel Tool Calls Enforcement: Added a post-parse fallback that caps chat completion responses to the first tool call when parallel_tool_calls: false is set, covering streaming and non-streaming responses across vLLM, SGLang, and TensorRT-LLM backends (#12201).
• Mistral Tokenizer Chat Template Fix: Fixed an AttributeError in the vLLM chat-processor path where MistralTokenizer lacks a chat_template attribute, which previously blocked Mistral models from registering and caused chat requests to return HTTP 404 (#12659).

Multimodal & Diffusion

• Preserved Media URL Client Errors: Fixed audio and video loaders to preserve URL-validation and HTTP-status errors during per-item and batch handling, so intentional client rejections keep their 4xx status instead of flattening into generic failures (#12407, #13193).
• GLM-Image NIXL Deploy Config: Migrated the GLM-Image AR to DiT NIXL disaggregation stage config off vllm-omni’s legacy stage_args schema to the stages deploy schema (#12984), following a fix that declared the entry stage’s engine_input_source after vllm-omni v0.26.0rc1 removed its default (#12709).
• Multimodal Trust-Remote-Code Fix: Fixed multimodal and encode workers across vLLM, SGLang, and TensorRT-LLM to thread trust_remote_code from the engine config instead of hardcoding True, so custom model repo code no longer executes on load unless the operator explicitly enabled it (#10738).
• SSRF-Blocked Media URL Status: Fixed multimodal image loading to return HTTP 400 instead of 500 when an image URL is blocked by the SSRF guard, preserving the UrlValidationError so clients get the block reason (#11312).

Scheduling

• SGLang Global Router Support: Fixed the prefill router’s execute_prefill logic to detect SGLang workers and return a Bootstrap outcome with bootstrap host, port, and room instead of hanging, so decode workers now receive requests when using SGLang with the global router (#7897).
• Planner Worker Discovery Hardening: Added runtimeNamespace to replica status so Planner reads the runtime namespace from DGD status (#11352), resolved workers by DGD component type (#11578), and matched model cards against .status.components[*].componentNames (#12047). GlobalPlanner read the v1beta1 spec.components schema (#11990) and rejected scaling with DynamoGraphDeploymentNotReadyError on unready deployments (#10526).
• Exact Push Router Dispatch: Fixed PushRouter::direct() to enforce exact-dispatch behavior, returning a typed CannotConnect error instead of silently rerouting to another worker when the selected instance disappears before transport resolution. Transport resolution now runs before overload checks, so stale overload metadata cannot mask a missing exact target (#10429).
• EAGLE3 KV Routing Fix: Fixed enable_eagle detection in SGLang to derive from SpeculativeAlgorithm.is_eagle() instead of a hand-maintained name set, correcting KV-aware routing for EAGLE3 and FROZEN_KV_MTP workers that previously reported zero cached blocks due to hash misalignment (#10982).
• KvRouter Cancellation Scope Fix: Fixed KvRouter to cancel a child token instead of the runtime’s primary token on drop, so rebuilding the Router after all workers exit no longer tears down the entire runtime with an etcd lease error (#11390).
• Prefill EAGLE KV Routing Fix: Fixed prefill KV routing to detect EAGLE/MTP support from the prefill endpoint’s runtime config instead of always disabling EAGLE hash mode, preventing zero cache-block predictions and restoring correct prefix cache hits when SGLang prefill workers enable EAGLE (#11764).

Kubernetes Deployment

• CRIU Checkpoint Image Mechanics: Added an imageIoMode CRIU knob (writeback or direct, defaulting to direct) and bumped go-criu to v8.4.0 (#11476), pinned the snapshot image CRIU build to a fixed revision (#11659), made rootfs diff capture atomic with zero-byte-archive rejection (#11883), and fixed the DRA nvidia-smi probe to enter the workload PID namespace (#12227).
• DGD Admission Webhook Migration: Served both v1alpha1 and v1beta1 DynamoGraphDeployment admission endpoints via the v1beta1 hub (#11117), restored pre-existing v1alpha1 behaviors with per-container CEL scoping (#11126), and fixed volume-mount composition so service PVC mounts survived (#11263). Made cluster-wide handlers Lease-aware (#11597) and gated Operator readiness on the webhook server with a TLS-checked /readyz (#12255).
• DGDR Profiling And Status Hardening: Added a reusable Go package that applies versioned overrides.dgd patches using Kubernetes structural merge semantics (#11124), an idempotent profiling-to-deployment handoff so a DynamoGraphDeploymentRequest no longer sticks in Deploying (#11250), and a MessageValidationPassed constant on the Validation=True condition (#11261).
• Explicit Service Mesh Override: Added a serviceMesh.enabled option to the Operator configuration so users can explicitly enable or disable Istio integration, fixing startup failures that occurred when Istio’s API group was partially present but the DestinationRule resource was unavailable (#10773).
• Docker Secret Index Resilience: Fixed the Operator’s controller manager crashing on startup when the initial Docker secrets index refresh encountered malformed kubernetes.io/dockerconfigjson secrets, now skipping invalid entries and logging aggregated errors while continuing to index valid ones (#10789).
• LWS Worker GPU Validation: Removed the hardcoded GPU limit check from LWS worker pod template generation, letting Kubernetes and device admission decide pod validity so CPU-only deployments no longer fail to render (#11115).
• Duplicate Compilation Cache Mounts: Fixed pod generation to reuse an existing writable compilation-cache mount instead of appending a duplicate one, and rejected mount-path conflicts and read-only cache mounts, preventing Grove PodCliqueSet validation failures on converted v1alpha1 components (#11264).
• EPP DiskPressure Resilience: Set default CPU, memory, and ephemeral storage requests for the EndPoint Picker (EPP) and lowered the default RUST_LOG level to info, preventing kubelet from evicting EPP under node DiskPressure and stalling InferencePool deployments (#11550).
• Multi-Node TensorRT-LLM Env Relay: Fixed multi-node TensorRT-LLM launch commands to forward the CPATH and TRITON_PTXAS_PATH environment variables and avoid duplicate forwarding, preventing build and compilation failures on multi-node deployments (#11999).
• DGDSA Replica Ownership Fix: Fixed a reconciliation race between the DynamoGraphDeployment and DynamoGraphDeploymentScalingAdapter (DGDSA) controllers by replacing full-spec DGDSA synchronization with a patch that preserves DGDSA-owned replica counts, preventing stale DGD values from overwriting scaling adapter replicas during serve, unserve, and scaling operations (#12087).
• Event Channel Metadata Compatibility: Fixed DynamoWorkerMetadata deserialization by reconstructing legacy namespace/component fields alongside the current scope field for Kubernetes event channels, preventing rolling upgrades from rejecting worker records due to version mismatches (#12708).

Fault Tolerance & Observability

• Bounded Endpoint Drain Shutdown: Fixed a zombie-worker failure mode where a lost etcd lease with a stuck in-flight request left a worker Running but unable to serve; per-endpoint drain now respects the graceful shutdown timeout and lease loss routes through Runtime::shutdown() (#11068).

Engines

vLLM

• Writable vLLM Cache Directories: Pre-created the vLLM cache root in the container image with group 0 write access so pods running under a UID other than 1000 with gid 0 no longer fail engine initialization with EACCES on the first torch compile (#13040), cherry-picked into the release branch (#13045).
• Lifecycle-Aware Disaggregated LoRA Prefill: Registered and served LoRA lifecycle endpoints on vLLM prefill workers, with metadata-only initial registration, adapters activated from the inference-time LoRARequest, and endpoint deduplication by physical pod (#11520). Fixed dynamic LoRA registration to publish adapter model cards with the engine-actual main-attention KV block size instead of the raw CLI block_size (#11867).
• vLLM Embedding Pooling Correctness: Set the embedding worker’s pooling task to embed in PoolingParams so vLLM returns one pooled, L2-normalized embedding per input (#10248), and propagated truncate_prompt_tokens to vLLM encode as tokenizer kwargs for raw-text embedding inputs (#11099).
• Embeddings Dimensions Type Validation: Fixed the /v1/embeddings endpoint to reject boolean values passed as the dimensions parameter, raising a TypeError instead of silently truncating embedding vectors to one dimension. Prevents corrupted embeddings from propagating undetected into downstream vector stores or RAG systems (#9873).
• Cache Salt Propagation Fix: Fixed the vLLM text-mode path to propagate nvext.cache_salt from the request into the prompt object sent to the vLLM engine, ensuring KV cache keys reflect the requested cache salt instead of losing it. (#11082).
• Qwen3-TTS Prompt Estimator Fix: Fixed Qwen3-TTS prompt-length estimation to call the estimator through Qwen3TTSPromptEmbedsBuilder instead of the stale talker-model import, avoiding the generic 2048-token fallback that could produce invalid or poor-quality audio (#11595).
• vLLM EngineCore Shutdown on Failed Startup: Fixed vLLM decode workers to shut down the EngineCore process when Dynamo-side initialization fails after engine startup, so failed workers exit cleanly instead of leaving a live engine process behind (#12867).

SGLang

• HiCache Capacity Publication: Fixed SGLang workers to publish ratio-based HiCache offload capacity, so Dynamo Router can account for SGLang CPU-offload cache tiers (#12953, #13060).
• Writable FlashInfer Cubin Directories: Marked the installed flashinfer_cubin/cubins directories writable in the SGLang runtime image before switching to the non-root dynamo user (#11154), added matching group-write permissions in the vLLM runtime images for arbitrary OpenShift UIDs in group 0 (#12770), and carried the vLLM fix into the release branch (#12976).
• SGLang Token Usage Reporting: Populated prompt_tokens_details.cached_tokens in terminal completion usage from final SGLang response metadata, restoring cache observability for warm requests including HiCache (#10925), and treated streamed usage metadata as optional so final chunks without token counts no longer raise KeyError (#11062).
• SGLang Response Format Forwarding: Fixed the SGLang chat processor to forward the OpenAI response_format parameter to decode and prefill engine calls, enabling SGLang-native JSON schema and JSON object structured output constraints when using --dyn-chat-processor sglang. (#10259).
• SGLang Multimodal Audio Forwarding: Fixed forwarding of Dynamo’s --enable-multimodal setting to SGLang’s server arguments and added extraction and forwarding of audio_url/audio_data alongside image and video inputs across aggregate, disaggregated prefill, and disaggregated decode paths. Unextracted audio content is now rejected using the same guard applied to other modalities (#11747).
• SGLang Bootstrap Host Discovery: Fixed prefill bootstrap endpoint discovery to prefer the concrete, non-loopback SGLang host from GetServerInfo before falling back to dist_init_addr or local-IP resolution. This prevents decode workers from receiving unreachable loopback or wildcard addresses for bootstrap (#12026).
• SGLang Conditional-Disagg Error Propagation: Fixed the SGLang rejection of the x-bypass-remote-prefill annotation so clients receive the intended unsupported-feature message instead of a bare Internal Server Error (#12616).
• SGLang NVExt Token Validation: Rejected out-of-vocabulary IDs from pre-tokenized nvext.token_data before reaching SGLang’s embedding lookup, while preserving valid request-scoped multimodal sentinel token IDs, backported to release/1.4.0 (#12852).

TensorRT-LLM

• TensorRT-LLM Mpmath Version Cap: Capped mpmath below 1.4 in the trtllm extra to prevent the resolver from pulling mpmath 1.5.0a0, which removed bitcount and caused the TensorRT-LLM backend to crash on import. (#11402).
• TensorRT-LLM Multimodal Request Handling: Scoped NIXL Connect initialization to multimodal disaggregated workers so plain text workers no longer load the nixl bindings (#11373), sized an omitted max_tokens from the expanded prompt length instead of the default 32 (#11513), and routed video_url inputs through Dynamo’s URL validation policy before local decoding (#12002).
• TensorRT-LLM RC22 Compatibility Fixes: Updated disaggregated serving for TensorRT-LLM rc22’s (node_id, process_id) machine-ID pair and stopped dispatching GEN when a context request finished during prefill without KV-cache handoff IDs (#12203), with engine argument defaults read through the changed TensorRT-LLM API (#11741).

Infrastructure Modernization

• ZMQ Subscriber Socket Leak: Fixed a leak where a dropped ZmqSubTransport left its subscriber pump running and SUB socket open, accumulating sockets until “Too many open files”. Pumps now stop when their final owner is dropped, and cancellation propagates to discovery watch tasks (#11730).

KV Block Manager

• KVBM Decode KV Cache Alignment: Configured KVBM prefill and decode workers with --disable-hybrid-kv-cache-manager so both sides share the same vLLM KV-cache layout, marking decode workers with --disaggregation-mode decode across launchers, examples, and the integration test (#12821). Stopped suppressing an explicit kv_events_config on disaggregated decode workers and set up the Dynamo KV-event relay for them (#12084).
• HMA Disable For PdConnector: Fixed KVBM and LMCache disaggregated serving startup failures by disabling vLLM’s hybrid KV cache manager (HMA) for PdConnector configurations whose child connectors, such as DynamoConnector or LMCacheConnectorV1 paired with NixlConnector, do not all support HMA, preventing the MultiConnector assertion during EngineCore startup (#11608).

General

• etcd Connection Resilience Fixes: Added a cancellable startup retry loop for the initial etcd connection and primary lease creation, retrying up to 120s with exponential backoff (#10799), and introduced a WatchEvent::Resync snapshot after watch reconnects so KvCache, TypedPrefixWatcher, and session-affinity claim watchers rebuild state and stale instances are removed (#11146).
• Shared ZMQ Context Fix: Fixed the event plane’s direct (broker-less) PUB/SUB transport to reuse a single process-wide ZMQ context instead of creating one per socket, preventing unbounded OS-thread growth on frontends as worker fleets scaled up, which had starved the async runtime and triggered liveness-probe restart loops (#11246).
• JSON5 Config Parsing Fix: Fixed max context length derivation to parse config.json with JSON5 instead of strict JSON, preventing registration failures for models like Nemotron-H whose configs contain non-finite literals such as Infinity in unused fields (#11445).
• Triton CUDA 13 Image Fix: Updated the Triton example’s default server image to tritonserver:25.10-py3 for CUDA 13.0.2 compatibility, fixed the DCGM library bind-mount path and permissions, and added a build-time import tritonserver smoke check (#12577).

Recipes

• DeepSeek-V4 H200 Checkpoint Download: Fixed the deepseek-v4 and deepseek-v4-pro H200 recipes to download the FP8 checkpoint instead of the B200 NVFP4 variant by parameterizing the model-download Job (#13034, #13213).
• DeepSeek-R1 Disagg Decode Mode: Fixed the deepseek-r1 vLLM disaggregated recipe to declare --disaggregation-mode decode on the decode worker, which previously never became ready (#13046).
• Removed Stale Router Flags: Removed the retired --router-reset-states flags from affected recipes so they launch cleanly against the shipped Router CLI (#13116, #13117).

Developer Experience

Documentation Website Refactor

• Tab-Based Documentation Navigation: Restructured the documentation site from a single flat sidebar into top-level Home, User Guide, Developer Guide, Recipes, Reference, Blog, and Community tabs, with an explicit redirect table of 357 redirects so every previously live URL still resolves (#10855). Restored the per-backend SGLang, TensorRT-LLM, and vLLM multimodal deployment guides to the navigation and fixed a stale anchor in their cross-references (#11064), and consolidated Kubernetes content while correcting remaining navigation, cross-reference, styling, and release and observability pages missed by the initial pass (#12243).
• Simplified Chinese Docs Localization: Registered all 13 existing zh-CN translations in the Fern nav under the hidden Additional Resources section so they publish at real URLs and the language switcher links resolve instead of serving soft-404 stubs (#11088), then replaced the raw HTML switcher badge with a plain Markdown link and dropped duplicate body H1s on 5 translated pages (#11148). Migrated the manual switcher to Fern native localization, moving translations to a fern/translations/zh-CN mirror tree with a translations: config that adds the header language dropdown and falls back to English for untranslated pages on the dev version (#11161).
• Tagged-Commit Release Snapshots: Fixed the Fern docs release workflow to build versioned pages and navigation from the tagged commit’s own docs, navigation, and converter instead of whatever content was on main, preventing release tags from publishing mismatched content, and added a local release dry-run script under fern/ for pre-publish validation (#11140).

Documentation

• Kubernetes & Deployment Docs: Restructured the Gateway API (GAIE) documentation into overview and quickstart pages with standalone EPP on-ramp guidance (#10956, #10993, #11119, #12480), corrected the Dynamo Operator’s default termination grace period (#11120), aligned install examples with current platform versions (#11486), and documented DGDR spec.overrides.dgd merge semantics (#11487), scale-to-zero behavior and its recovery limits (#11536), and the TensorRT-LLM snapshot support matrix (#11674).
• Recipe & Model Pages: Added recipe pages for the Inkling NVFP4 day-0 launch (#11749, #11807), GLM-5.2 on SGLang across B200 NVFP4 and H200 FP8 targets (#11972), Kimi-K3 (#12210), and vLLM agentic targets for GPT-OSS-120B (#11980), plus a Model Early Access Builds reference page standardizing early-access terminology (#11898).
• Router & KV Routing Reference: Corrected the Router reference pages to match shipped v1.4.0 behavior (#11892, #12949, #12983), clarified that KV routing requires explicit worker-side event publishing and distinguished the fallback modes (#11505), added a KV offloading support matrix across backends and cache tiers (#11973), updated SGLang HiCache minimum-version requirements (#11183), and corrected session-radix ownership claims (#11241).
• Backend Setup & Troubleshooting: Directed TensorRT-LLM users to the container-based install path (#11066), recommended the PyTorch multiprocessing backend over Ray for vLLM multi-node deployments (#11259), documented the CUDA 13 FlashInfer sampler failure and its workaround (#10954), the NVENC hardware requirement for TensorRT-LLM video diffusion MP4 output (#11457), corrected the TensorRT-LLM UCX environment-variable note and EFA image facts (#13049), vLLM worker-role flags and --kv-transfer-config (#12666), and XPU deployment paths for Intel GPUs including DRA requirements (#11894).
• Planner & Profiler Docs: Documented rapid-profiling behavior when no SLA-feasible configuration exists and corrected the unachievable-SLA constraints claim (#11612, #12774), and corrected the Planner configuration reference covering all four optimization targets and TLS verification behavior (#11613).
• Simulation Docs: Reframed the DynoSim documentation around deterministic offline replay and live Mocker with a dedicated simulation-model page (#12172, #11430), corrected the load-generator ISL/OSL preset semantics (#10791), and corrected Spica’s kv_load_ratio support guidance (#12714).
• Observability Docs: Documented the actual OTLP endpoint resolution chain and sampling controls, removing a stale fallback claim that caused silent log loss (#11337), and corrected tokenizer-cache metric absence semantics while adding the v1.4.0 deprecations section (#12775), and marked the CKF counter as registered but never incremented in this release (#13149).
• Cleanup: Replaced references to the removed experimental unified backend engines with current guidance (#11833).

Looking Ahead

Standalone Router & Runtime-Free EPP

The standalone Router work continues toward a runtime-free selection service and inference-gateway EPP that deploy without the full Dynamo runtime, extending the endpoint-scoped transport and reservation-replay foundations shipped in this release. The Rust ext-proc EPP becomes the default gateway integration in an upcoming release, replacing the Go-based Endpoint Picker.

Custom Routing Strategies

Router scoring and selection open up behind native WorkerScore and WorkerPick policies already merged on main, so deployments can bring a routing strategy tuned to their workload. The trace-capture and simulation tooling shipped in this release gives those strategies a place to be validated against recorded production traffic before they route real requests.

Cross-Datacenter Routing

Cross-datacenter prefix routing shipped experimental in Dynamo v1.4.0. The work ahead hardens the datacenter-scoped KV relay and sequenced indexing toward production use, and workers move toward advertising their own router configuration so fleets negotiate routing behavior without frontend flag coordination.

Voice & Custom Multimodal Encoders

Multimodal work extends toward custom encoder support, frontend-decoded encode-prefill-decode pipelines, and realtime voice serving, building on the unified vLLM multimodal serving and transcription infrastructure in this release.