跳到导航
Image UnderstandingVLM Performance Optimization

Shared Image Cache

Reuse downloaded image bytes across workers
以 Markdown 格式查看

The optional shared cache stores encoded bytes from HTTP and HTTPS image URLs before image decoding. Workers can reuse those bytes across processes while keeping their local decoded-image caches independent. Data URLs, frontend-decoded NIXL inputs, video, and audio do not use this cache.

Images seen before are found in the shared encoded-image cache and returned without a download. A new image is a miss: it is downloaded once from the image origin and stored in the cache for the next worker.

Backend Support

BackendModeShared encoded-image cache
vLLMAggregated and disaggregated paths that load URLs through Dynamo ImageLoaderSupported
TensorRT-LLMAggregated and disaggregated paths that load URLs through Dynamo ImageLoaderSupported
SGLangStandard URL loadingNot supported. Dynamo passes URL strings directly to SGLang.
SGLang--frontend-decodingOnly URL items that still reach Dynamo ImageLoader can use the cache. NIXL-decoded items bypass it.

Dynamo does not provision or discover a cache service. Deploy Redis Cluster or Dragonfly separately and provide a cluster endpoint that every participating URL-loading worker can reach. The connection URL carries the endpoint, authentication, and TLS choice. See Cache Service Deployment for single-node and multi-node layouts.

Set the following environment variables on every participating worker:

Environment variableDefaultDescription
DYN_MM_SHARED_IMAGE_CACHE_ENABLED0Set to 1 to enable the shared cache.
DYN_MM_SHARED_IMAGE_CACHE_URLNoneRedis Cluster or Dragonfly connection URL. Required when the cache is enabled.
DYN_MM_SHARED_IMAGE_CACHE_TTL_SECS3600Positive cache-entry lifetime in seconds.
DYN_MM_SHARED_IMAGE_CACHE_CONNECT_TIMEOUT_SECS0.1Positive connection timeout in seconds.
DYN_MM_SHARED_IMAGE_CACHE_IO_TIMEOUT_SECS2.0Positive Redis operation timeout in seconds.
DYN_MM_MAX_FILE_SIZE_MB64Maximum encoded image size downloaded from the origin, in MiB. The limit applies whether or not the shared cache is enabled, so oversized images are never stored in it.
DYN_MM_IMAGE_CACHE_SESSION_SCOPED0Set to 1 to partition Dynamo image and image-embedding caches by session affinity.

Store the full connection URL in a Kubernetes Secret and inject it into each worker rather than placing credentials in a manifest:

env:
  - name: DYN_MM_SHARED_IMAGE_CACHE_ENABLED
    value: "1"
  - name: DYN_MM_SHARED_IMAGE_CACHE_URL
    valueFrom:
      secretKeyRef:
        name: multimodal-image-cache
        key: connection-url
  - name: DYN_MM_IMAGE_CACHE_SESSION_SCOPED
    value: "1"

Use a rediss:// URL unless Redis traffic is protected by equivalent transport encryption, such as an encrypted service mesh. A redis:// URL sends cache data and any URL credentials without TLS and should be used only on an appropriately secured in-cluster network.

Cache reads and fills run on the request path. A miss waits for Redis SET so the fill completes before the request returns. Redis errors fail open, and each cache operation is attempted once. While a cache node is unreachable, an image load pays one failed GET and one failed SET: each is bounded by DYN_MM_SHARED_IMAGE_CACHE_CONNECT_TIMEOUT_SECS when a new connection is refused or times out, and by DYN_MM_SHARED_IMAGE_CACHE_IO_TIMEOUT_SECS when an open connection stops responding. Warnings are emitted initially and at most once per minute; individual operation failures remain available at debug level. Dynamo does not currently use a bounded asynchronous write queue.

Cache Service Deployment

Dynamo uses the Redis Cluster protocol for both backends. The client uses DYN_MM_SHARED_IMAGE_CACHE_URL only to discover the cluster: it reads the slot map from that endpoint and then connects to each node at the address the node announces. When a node stops responding, the client rediscovers through the same URL, so point it at an address that survives cache restarts, such as a Kubernetes Service that selects every cache node, not at an individual pod. After a restart, the nodes announce their new addresses and workers pick them up without restarting. Dynamo requires redis-py 6.2 or later.

Single-node Dragonfly. Run Dragonfly with --cluster_mode=emulated so that it answers Redis Cluster commands, and keep its default announced address (the pod IP). Do not set --cluster_announce_ip to the host in DYN_MM_SHARED_IMAGE_CACHE_URL: with redis-py 6.2 through 7.1, a node that announces the same host and port as the URL removes the URL from the client’s rediscovery list after one failed operation.

Redis Cluster. Run the nodes as a StatefulSet with a headless Service for per-pod DNS names, plus a regular Service that selects every node for the URL. Configure each node with:

  • cluster-announce-ip set to the pod IP, or cluster-announce-hostname set to the pod’s DNS name together with cluster-preferred-endpoint-type hostname (Redis 7.0 or later).
  • cluster-require-full-coverage no, so the remaining nodes keep serving while one node is down. Images that hash to the unavailable node fall back to the origin.
  • A nodes.conf that persists across pod restarts, so a restarted node keeps its identity and slots. A rolling restart then keeps the cluster available. If every node restarts at once, all peer addresses change and the nodes need their peers’ new addresses before they can re-form the cluster.

Multi-node Dragonfly. With --cluster_mode=yes, Dragonfly nodes do not discover each other. A cluster orchestrator must push the slot map with DFLYCLUSTER CONFIG and push it again whenever a node restarts, because the map is kept in memory. Point the URL at a Service that selects every node, as for Redis Cluster.

Session Scoping

Enable DYN_MM_IMAGE_CACHE_SESSION_SCOPED=1 when the same URL can resolve to different content for different sessions. Dynamo derives the scope from these headers, in precedence order:

  1. x-dynamo-session-id
  2. Recognized agent headers: Claude Code x-claude-code-session-id (or x-claude-code-agent-id for a child agent), Codex thread-id, then OpenCode x-session-id

Use x-dynamo-session-id to provide an explicit Dynamo session identity. Dynamo normalizes it and the recognized agent headers into both an AgentContext and the session affinity used for routing and image-cache scoping.

Session scoping partitions cache entries; it is not an authentication boundary. Unless a trusted proxy sets or overwrites x-dynamo-session-id, the caller controls the scope. Configure the proxy to derive this header from the authenticated identity on every request, and treat scope identifiers as secrets. A caller that learns another identifier can reuse cached content stored under that scope for the same URL.

When session scoping is enabled, a missing or blank scope bypasses the local decoded-image cache, in-flight request deduplication, the shared encoded-image cache, and Dynamo-owned image embedding caches. A valid scope partitions all of those caches. The scope does not partition backend KV cache keys; affinity can route a session back to the same worker, but KV identity remains content-based.