Using HiCache

This guide covers running SGLang’s Hierarchical Cache (HiCache) with Dynamo, and how the Dynamo KV router integrates with HiCache for tier-aware worker selection when workers share an external pool such as Mooncake.

Overview

SGLang HiCache extends RadixAttention with a multi-tier KV cache that transparently moves pages between GPU HBM, host memory, and an optional external storage backend (e.g. Mooncake). For a full description of HiCache itself — flag reference, storage backends, memory layouts, prefetch policies — see SGLang’s own documentation:

• SGLang HiCache Design
• SGLang HiCache Best Practices

What Dynamo adds on top of HiCache:

• Tier-aware routing. The KV router tracks which cache tier each block lives on (GPU / Host / External) and uses that when scoring candidate workers — not just device overlap.
• Shared-pool awareness. When an external backend such as Mooncake is configured, the router tracks Mooncake object events so it can discount prefill cost for blocks any worker can fetch, not just blocks the candidate holds locally.

If you are running a single worker with HiCache and no shared pool, no Dynamo-side configuration is required — the worker reports KV events to the router as usual.

Running SGLang with HiCache

Launch a worker with HiCache enabled:

$
python -m dynamo.sglang \
>
  --model-path Qwen/Qwen3-0.6B \
>
  --page-size 64 \
>
  --enable-hierarchical-cache \
>
  --hicache-ratio 2 \
>
  --hicache-write-policy write_through \
>
  --hicache-storage-backend nixl \
>
  --skip-tokenizer-init

Then start the frontend:

$
python -m dynamo.frontend --http-port 8000

Tier-Aware Shared KV Cache Routing

When you scale out to multiple SGLang workers that share an external pool such as Mooncake, the Dynamo router can be made tier-aware. It tracks per-tier residency from worker events and consults the shared pool directly so that blocks cached anywhere in the cluster — not just on the candidate worker’s GPU — contribute to worker scoring.

Why

By default the router’s radix tree only reflects blocks resident in GPU HBM on each worker. HiCache silently demotes blocks to host memory and further to Mooncake as the device pool fills, but the router never sees those transitions. A worker that has the full request prefix on host + Mooncake looks identical to a cold worker. The router ends up treating “fetchable from Mooncake in milliseconds” the same as “must be recomputed from scratch.”

Event model

SGLang’s HiRadixCache emits BlockStored / BlockRemoved events carrying a medium field on every tier transition:

CPU_PINNED is the value SGLang’s HiRadixCache actually emits for host-tier blocks (page-locked memory). The rest of this guide uses CPU_PINNED to match the on-the-wire string; “Host” is the conceptual tier name.

A few properties the router relies on:

• Ordering. store(new_tier) is emitted before remove(old_tier) so the block is never invisible to the router during a transition.
• DMA safety. store(CPU_PINNED) for a GPU→Host copy is deferred until finish_event.synchronize() confirms the DMA landed — events never fire before bytes are resident.
• Per-tier tracking. A block can be on GPU and Host simultaneously. The router records both and picks the highest-priority tier when scoring overlap.

How it works

The router maintains two indexes:

• Its own radix tree, built from worker KV events (per-tier).
• A set of Mooncake object keys and verified logical groups, built from the Mooncake master’s event stream.

Request routing checks both indexes locally. When an event includes SGLang’s group_id, the first lookup verifies every physical object in the group and caches the result. Later requests use one group lookup per page; any event for that group invalidates the cached result. Events without group_id use exact object-key checks.

If the event stream starts after Mooncake already contains objects, the shared-pool index starts empty and learns about subsequent events. A sequence gap clears the shared-pool index to avoid stale hits.

Scoring

For each candidate worker, the router computes a logit (lower wins):

# Without shared cache
adjusted_prefill_blocks = max(
    prefill_blocks
    - overlap_score_credit * device_overlap_blocks
    - host_cache_hit_weight * host_overlap_blocks
    - disk_cache_hit_weight * disk_overlap_blocks,
    0,
)
logit = prefill_load_scale * adjusted_prefill_blocks + decode_blocks
# With shared cache
shared_beyond = shared_cache_hits.hits_beyond(device_overlap_blocks)
adjusted_prefill_blocks = max(
    prefill_blocks
    - overlap_score_credit * device_overlap_blocks
    - host_cache_hit_weight * host_overlap_blocks
    - disk_cache_hit_weight * disk_overlap_blocks
    - shared_cache_multiplier * shared_beyond,
    0,
)
logit = prefill_load_scale * adjusted_prefill_blocks + decode_blocks

hits_beyond(n) counts shared-cache pages at positions >= n — “pages past my device prefix that I can still fetch from Mooncake instead of recomputing.”

Worked example. Request is 4 blocks, shared_cache_multiplier = 0.5, block_size = 1, overlap_score_credit = 1.0 (the maximum device-local overlap credit). Shared pool contains blocks 0–3.

W0 wins because it combines device-local reuse with shared-pool hits beyond that device prefix. The multiplier encodes the cost ratio of a Mooncake fetch relative to a fresh GPU compute — 0.5 means “fetching from shared is half as expensive as recomputing.”

Requirements

Earlier SGLang versions do not emit medium=CPU_PINNED for Host-tier residency, so the router can only track GPU events regardless of Mooncake configuration.

You also need:

• Dynamo router started with --shared-cache-type hicache (see Configuration).
• A Mooncake master with KV events enabled. This requires the event publisher introduced by kvcache-ai/Mooncake#2214 until that change is available in a Mooncake release.
• A Mooncake KV event endpoint reachable from the Dynamo frontend host. Worker-side Mooncake config (event endpoint, page size, TP/PP layout, and split-head layout) is published automatically through each worker’s registration metadata.

Setup

SGLang worker — HiCache with Mooncake storage:

$
python -m dynamo.sglang \
>
  --model-path Qwen/Qwen3-0.6B \
>
  --page-size 64 \
>
  --enable-hierarchical-cache \
>
  --hicache-ratio 2 \
>
  --hicache-write-policy write_through \
>
  --hicache-storage-backend mooncake \
>
  --hicache-storage-backend-extra-config '{"master_server_address": "mooncake-master.internal:50051", "enable_group_semantics": true}' \
>
  --skip-tokenizer-init

Launch additional workers on other GPUs / hosts with the same Mooncake config so they back to the same cluster.

Dynamo frontend — configure the Mooncake event endpoint and enable tier-aware routing:

$
DYN_MOONCAKE_KV_EVENTS_ENDPOINT=tcp://mooncake-master.internal:5557 \
>
python -m dynamo.frontend \
>
  --http-port 8000 \
>
  --router-mode kv \
>
  --shared-cache-type hicache \
>
  --shared-cache-multiplier 0.5

Configuration

Per-request overrides are available via RouterConfigOverride.shared_cache_multiplier for A/B experimentation without restarting the router.

Set DYN_MOONCAKE_KV_EVENTS_ENDPOINT on the frontend to the Mooncake PUB endpoint, such as tcp://mooncake-master.internal:5557. The endpoint must be reachable from the frontend. Workers can advertise the same variable as a fallback, but a missing worker value does not disable shared-cache routing.

Set enable_group_semantics to true in --hicache-storage-backend-extra-config to include SGLang logical group IDs in Mooncake metadata. Dynamo falls back to exact physical-key checks when group metadata is unavailable.

Verification

Events carry a medium. Run the worker with --log-level debug and grep the log:

$
python -m dynamo.sglang ... --log-level debug 2>&1 | grep -E 'BlockStored|BlockRemoved'
$
# BlockStored(block_hashes=[...], medium=CPU_PINNED)
$
# BlockRemoved(block_hashes=[...], medium=GPU)

If medium is missing or Host-tier transitions never report CPU_PINNED, confirm that the worker runs SGLang 0.5.11 or later (or a custom build that includes PR #22894).

Router sees the shared pool. Shared-cache metrics are exposed on the frontend’s Prometheus endpoint:

$
curl -s localhost:8000/metrics | grep shared_cache

Troubleshooting

Further Reading

• SGLang HiCache Design and Best Practices
• Mooncake — the shared KV store used as the external tier
• SGLang PR #22894 — the tier-annotated events prerequisite
• KVBM Guide — Dynamo’s own block manager, an alternative to HiCache
• KV Events for Custom Engines — the event protocol contract for backends other than SGLang