DynoSim Simulation Model | NVIDIA Dynamo Documentation

DynoSim builds a distributed serving simulation from Mocker engine cores. Each core owns engine-specific scheduler and KV-cache state. Offline prediction and live Mocker workers drive the same core through virtual-time and wall-clock execution, respectively.

Run python3 -m dynamo.mocker to launch live workers. This worker launcher is separate from the former public Replay online CLI, which remains unavailable in the unified AISimulate CLI. The Python replay SDK retains online mode for programmatic callers.

Engine Behavior

Mocker exposes three engine modes through two scheduler cores. The TensorRT-LLM mode uses the vLLM-shaped core with a different capacity policy.

Behavior	vLLM	SGLang	TensorRT-LLM
Scheduler model	Waiting and running queues with a shared token budget	Cache-aware waiting and running queues	vLLM-shaped core with capacity-first admission
KV representation	Native block pool	Token pool and radix cache	Native block pool
Memory pressure	LIFO or FIFO recompute preemption	Decode retraction with cached-prefix preservation	`GUARANTEED_NO_EVICT`; reserves prompt plus maximum output at admission
Prefix reuse	Block-hash matching when prefix caching is enabled	Radix-prefix matching	Block-hash matching when prefix caching is enabled
Default block or page size	64 tokens	1 token, or `sglang.page_size`	32 tokens
Aggregated simulation	Supported	Supported	Supported
Prefill/decode disaggregation	Supported	Supported	Not supported

Data-parallel ranks own independent scheduler and KV-pool state. The internal worker runtime and offline prediction compose those ranks into one logical worker with a shared pass barrier.

KV Managers

The shared vLLM/TensorRT-LLM core uses Mocker’s self-contained physical block pool to model GPU KV capacity, prefix reuse, request ownership, least-recently-used eviction, and router-visible KV events. The SGLang core uses its own token-pool and radix-cache implementation.

Timing Sources

Scheduler state determines the batch and cache-hit inputs to the timing source. Choose one timing source for each worker role with engine.workers.<role>.timing.type.

The live Mocker CLI options for profile-derived interpolation and direct --aic-* configuration are separate from the unified CLI. In AISimulate configurations, use only the timing types accepted by the AISimulate YAML schema.

AISimulate Performance Model

timing.type: default uses the AISimulate performance model. AISimulate selects model data from engine.model, engine.hardware, engine.backend, the optional engine.backend_version, and the role’s parallelism mapping. Unsupported combinations fail instead of silently using an uncalibrated timing model.

KV-capacity selection is independent of the timing source. With kv_cache.capacity.type: default, AISimulate estimates capacity from the model, backend, hardware, parallelism, block size, and memory fraction. With type: fixed, set the concrete block count in blocks.

Fixed Timing

Set timing.type: fixed and provide both prefill_ms and decode_ms to apply constant durations:

1 timing: {type: fixed, prefill_ms: 2, decode_ms: 0.5}

Use fixed timing for functional tests or controlled comparisons. These values replace the AISimulate timing lookup but do not disable default KV-capacity estimation.

Polynomial Baseline

Set timing.type: polynomial to use the uncalibrated synthetic baseline. Prefill latency follows a polynomial over the uncached tokens scheduled in the pass. Decode latency follows a polynomial over active KV-cache utilization.

Prefill/Decode Handoff

For engine.mode: disaggregated, define engine.workers.prefill and engine.workers.decode. Configure the modeled transfer under engine.kv_transfer:

1 kv_transfer:
2   bytes_per_token: auto
3   bandwidth_gb_per_second: 400
4   timing_mode: destination_missing

The offline simulation uses a per-request line-rate model:

transfer time = transferred KV bytes / kv_transfer_bandwidth
transferred KV bytes = charged tokens * kv_bytes_per_token

timing_mode: full_prompt charges the full logical prompt. destination_missing charges only the prompt footprint absent from the destination and can produce zero delay on a full destination hit. Concurrent requests do not contend for a shared link.

Set bytes_per_token: auto to derive the value from model metadata and parallelism, or provide a positive integer. Set a positive bandwidth_gb_per_second to apply the transfer delay. TensorRT-LLM does not support disaggregated simulation.

Distributed Signals

The internal worker runtime publishes the same categories of signals consumed by the distributed runtime:

stored and removed KV events for KV-aware routing;
engine-shaped Prometheus scheduler and request metrics;
per-rank cache, queue, running-request, and preemption counters;
forward-pass metrics with scheduled and queued prefill/decode work.

Offline KV-router replay captures the corresponding scheduler events and applies them to an in-process indexer at deterministic event boundaries.

Fidelity Boundaries

Interpret simulation results within these boundaries:

Timing accuracy depends on the selected timing source and its calibration range.
KV capacity and state transitions are modeled; KV tensor payloads are not.
The internal worker runtime includes Dynamo component and transport overhead but does not measure GPU kernel or inference-engine overhead.
Offline replay replaces external services and wall-clock concurrency with an event queue and shared logical clock.
TensorRT-LLM disaggregation and multi-tier KV offload are not modeled.
Mocker simulates text-token processing; it does not model multimodal encoder or cross-attention compute.

Use offline prediction for broad algorithm and configuration exploration. Use live Mocker workers to exercise the distributed path, and validate performance conclusions with focused GPU benchmarks.