Mocker CLI Reference | NVIDIA Dynamo Documentation

python3 -m dynamo.mocker launches a simulated Dynamo worker that registers with the frontend, publishes KV events, and exercises the router and planner paths without a GPU. This page is the flag reference. For the deployment workflow and local/Kubernetes launch recipes, see Simulate a Kubernetes Deployment or Simulate a Local Deployment; for the engine internals these flags configure, see Mocker Engine Architecture.

Run python3 -m dynamo.mocker --help for the complete option list supported by the installed version.

AISimulate reuses the same engine core for offline virtual-clock prediction and recommendation, but its public YAML is separate from these live-worker flags. See the DynoSim Replay CLI Reference for the offline contract. The former public Replay online CLI remains unavailable, although the Python replay SDK retains online mode.

Core and model

--version

booleanDefaults to false

Print the Dynamo Mocker version and exit.

--model-path

stringDefaults to null

Hugging Face model ID or local path for the tokenizer. Set this for normal frontend requests.

--model-name

stringDefaults to derived from --model-path

Model name used in API responses.

--endpoint

stringDefaults to auto-derived

Dynamo endpoint string. Defaults are namespace-dependent, and prefill workers use a different default endpoint than aggregated or decode workers.

--engine-type

stringDefaults to vllm

Engine simulation type.

Allowed values: vllm sglang trtllm

--extra-engine-args

pathDefaults to null

Path to a JSON file with canonical AISimulate launch configuration and optional dynamo runtime options. Engine fields belong under engine. This file overrides individual CLI arguments; see Canonical Python Engine Configuration.

KV cache

--num-gpu-blocks-override

integerDefaults to auto

Usable KV cache blocks per data-parallel rank. Non-AIS timing defaults to 16,384 blocks; AIS timing estimates capacity when this option is omitted.

--block-size

integerDefaults to engine-specific

Tokens per KV cache block. Defaults are 64 for vLLM, 1 for SGLang, and 32 for TensorRT-LLM.

--max-model-len

integerDefaults to null

Maximum sequence length, including prompt and generated tokens. Omitting the option leaves the simulated engine without a model-length limit.

--enable-prefix-caching

booleanDefaults to true

Enable prefix caching. Pass --no-enable-prefix-caching to disable.

--kv-cache-dtype

stringDefaults to auto

KV cache dtype for the bytes-per-token computation.

--kv-bytes-per-token

integerDefaults to auto-computed

KV cache bytes per token. Overrides the auto-computation.

Native G2 host offload

The vLLM mocker can simulate framework-native G1-to-host (G2) KV offload. Offloaded blocks are restored from host memory on a later prefix hit. As with the native vLLM offloading connector, G2 residency is published as HostPinned (CPU-medium) KV events alongside device events, so the KV router can credit a host-resident prefix with host_cache_hit_weight and route to the worker that holds it. Host offload requires --engine-type vllm and prefix caching. Host blocks are sized by --kv-bytes-per-token.

G2 events are not published on the vLLM ZMQ wire (--zmq-kv-events-ports), which needs per-block token IDs that the simulated G2 does not carry; a router consuming that stream sees only G1.

--num-host-blocks

integerDefaults to null

Enable native G2 host offload with this host cache capacity per data-parallel rank, in blocks.

--host-offload-d2h-bandwidth-gbps

floatDefaults to AISimulate default

Per-rank device-to-host bandwidth in decimal GB/s. 0 is unlimited. Requires --num-host-blocks.

--host-offload-h2d-bandwidth-gbps

floatDefaults to AISimulate default

Per-rank host-to-device bandwidth in decimal GB/s. 0 is unlimited. Requires --num-host-blocks.

Offline replay can also simulate one G2 pool shared by every worker of the simulated deployment. Live mocker workers reject this scope, so pass it through the offline replay Python API:

from dynamo.replay import run_trace_replay
engine_args = {
    "engine": {
        "backend": "vllm",
        "kv_cache_bytes_per_token": 131072,
        "native_host_offload": {
            "scope": "cluster_shared",
            "num_host_blocks": 65536,
            "kv_layout_id": "llama-3-8b-tp1",
        },
    },
}
report = run_trace_replay(
    "trace.jsonl",
    extra_engine_args=engine_args,
    num_workers=4,
    replay_mode="offline",
    router_mode="kv_router",
)

Offline replay does not compute bytes per token from a model, so set engine.kv_cache_bytes_per_token explicitly. engine.native_host_offload also accepts latency_to_first_byte_ms and the shared-pool bandwidth caps. G2 host offload does not support MTP (--ais-nextn).

Scheduling

--max-num-seqs

integerDefaults to 256

Maximum concurrent sequences.

--max-num-batched-tokens

integerDefaults to 8192

Maximum tokens per batch.

--enable-chunked-prefill

booleanDefaults to true

Enable chunked prefill. Pass --no-enable-chunked-prefill to disable.

--preemption-mode

stringDefaults to lifo

Decode eviction policy under memory pressure. lifo is vLLM v1 style.

Allowed values: lifo fifo

Timing

--speedup-ratio

floatDefaults to 1.0

Timing speedup factor applied to simulated prefill and decode durations.

--decode-speedup-ratio

floatDefaults to 1.0

Decode-only speedup multiplier, for example for Eagle speculation.

--startup-time

floatDefaults to null

Simulated startup delay, in seconds.

Data parallelism and workers

--data-parallel-size

integerDefaults to 1

Number of DP replicas.

--num-workers

integerDefaults to 1

Workers per process. Prefer this over launching many separate mocker processes: all workers share one tokio runtime and thread pool.

--stagger-delay

floatDefaults to 0

Delay between worker launches, in seconds. 0 disables staggering (default); -1 enables auto mode.

Performance modeling

--planner-profile-data

pathDefaults to null

Path to either a mocker-format .npz file or a profiler results directory.

--reasoning

JSON stringDefaults to null

JSON config for emitting reasoning token spans, with start_thinking_token_id, end_thinking_token_id, and thinking_ratio.

--response-replay-trace-path

pathDefaults to null

Mooncake JSONL trace whose output_token_ids provide response replay annotations.

AIS performance model

--ais-perf-config

JSON object or JSON/YAML fileDefaults to null

Complete AISimulate ForwardPassPerfModelConfig, including immutable worker role, data roots, selection policy and nested estimator controls. Defaults to estimation_mode: auto and fallback_policy: deny. Do not combine this input with flat AIS identity flags. Cold regression is rejected by Router and Mocker. Mocker and Dynamo Replay require pp: 1; pipeline-parallel configurations are rejected.

Legacy --aic-* spellings remain input aliases. Use --ais-* in new configurations.

Opt-in flags for the AISimulate (AIS) latency model. Live Mocker workers do not use AIS by default. For the timing-model design, see Mocker Engine Architecture.

--ais-perf-model

booleanDefaults to false

Use the canonical AISimulate performance-model API for latency prediction instead of the interpolated or polynomial models. Install aisimulate and use a supported system/backend/version tuple.

--ais-system

stringDefaults to h200_sxm when AIS is enabled

AIS system name used with --ais-perf-model. The raw CLI value is unset when omitted; runtime resolution uses h200_sxm when AIS modeling is enabled.

--ais-backend

stringDefaults to --engine-type

Backend used for AIS performance-data lookup. Set it only to model timing for a different backend than the simulated scheduler.

Allowed values: vllm sglang trtllm

--ais-backend-version

stringDefaults to auto

AIS performance-database version. Use current, previous, or next when the slot is available for the selected system and backend, or specify a version assigned to one of those slots. When unset, uses the release database’s current slot.

--ais-tp-size

integerDefaults to 1

Tensor-parallel size for AIS latency prediction. Affects only AIS performance-model lookups, not mocker scheduling.

--ais-moe-tp-size

integerDefaults to null

Mixture-of-Experts tensor-parallel size for AIS latency prediction. Required by some MoE models.

--ais-moe-ep-size

integerDefaults to null

Mixture-of-Experts expert-parallel size for AIS latency prediction. Required by some MoE models.

--ais-attention-dp-size

integerDefaults to null

Attention data-parallel size for AIS latency prediction. Required by some MoE models.

--gpu-memory-utilization

floatDefaults to 0.9

vLLM GPU memory fraction used for AIS KV-capacity estimation.

--mem-fraction-static

floatDefaults to 0.88

SGLang static memory fraction used for AIS KV-capacity estimation.

--free-gpu-memory-fraction

floatDefaults to 0.9

TensorRT-LLM fraction of post-model-load free GPU memory used for AIS KV-capacity estimation.

--ais-nextn

integerDefaults to null

Experimental Multi-Token Prediction (MTP) draft length, from 1 through 5.

--ais-nextn-accept-rates

stringDefaults to null

Experimental comma-separated conditional MTP acceptance rates.

--ais-mtp-seed

integerDefaults to 42

Base random seed for Mocker MTP burst sampling.

Disaggregation

--disaggregation-mode

stringDefaults to agg

Worker mode.

Allowed values: agg prefill decode

--is-prefill-worker

booleanDefaults to false

Deprecated alias for --disaggregation-mode=prefill. Do not combine it with --disaggregation-mode or --is-decode-worker.

--is-decode-worker

booleanDefaults to false

Deprecated alias for --disaggregation-mode=decode. Do not combine it with --disaggregation-mode or --is-prefill-worker.

--bootstrap-ports

stringDefaults to null

Comma-separated rendezvous base ports, one per worker in disaggregated mode.

--kv-transfer-bandwidth

floatDefaults to null

KV cache transfer bandwidth in GB/s. Set to 0 to disable.

--kv-transfer-timing-mode

stringDefaults to full_prompt

Prompt footprint charged to a coordinated prefill/decode transfer.

Allowed values: full_prompt destination_missing

Event and request transport

--event-plane

stringDefaults to auto

Event transport. When the environment does not set a value, Mocker uses ZMQ with file or memory discovery and NATS with etcd or Kubernetes discovery.

Allowed values: nats zmq

--request-plane

stringDefaults to env-driven (tcp)

Request transport.

Allowed values: nats tcp

--discovery-backend

stringDefaults to env-driven (etcd)

Discovery backend.

Allowed values: kubernetes etcd file mem

--zmq-kv-events-ports

stringDefaults to null

Comma-separated ZMQ PUB base ports for KV event publishing, one per worker.

--zmq-replay-ports

stringDefaults to null

Comma-separated ZMQ ROUTER base ports for gap recovery, one per worker.

SGLang-specific

Apply only when --engine-type sglang. Set the radix-cache page size with --block-size.

--sglang-schedule-policy

stringDefaults to fifo

SGLang scheduling policy. fifo is the default; lpm is longest prefix match. The deprecated fcfs alias maps to fifo and prints a warning when used.

Allowed values: fifo lpm

--sglang-page-size

integerDefaults to null

Deprecated alias for --block-size when --engine-type sglang. Prints a warning when used; other backends ignore it. If both flags are supplied for SGLang, their values must match.

Both aliases are deprecated in Dynamo 1.6.0 and will be removed in Dynamo 1.8.0, after two releases. Use --sglang-schedule-policy fifo and --block-size instead.

--sglang-max-prefill-tokens

integerDefaults to 16384

SGLang max prefill-token budget per batch.

--sglang-chunked-prefill-size

integerDefaults to 8192

SGLang chunked-prefill chunk size.

--sglang-clip-max-new-tokens

integerDefaults to 4096

SGLang admission-budget cap for max new tokens.

--sglang-schedule-conservativeness

floatDefaults to 1.0

SGLang schedule conservativeness factor.

TensorRT-LLM-specific

Apply only when --engine-type trtllm.

--trtllm-capacity-scheduler-policy

stringDefaults to guaranteed_no_evict

TensorRT-LLM capacity scheduler policy. The Mocker currently supports only guaranteed_no_evict.

Router advertisement

Mocker accepts the same --router-* worker-advertisement options as the real backend launchers. Use them to override Router behavior for this worker set. See Router Configuration and Tuning for the shared fields.

--router-mode

stringDefaults to inherit frontend

Advertise a Router configuration for this worker set. When omitted, the worker advertises no Router configuration and inherits the Frontend configuration.

Allowed values: round-robin random power-of-two kv direct least-loaded device-aware-weighted

Environment variables

Variable	Default	Description
`DYN_MOCKER_KV_CACHE_TRACE`	off	Set to `1` or `true` to log structured native KV-cache allocation and eviction traces.

Simulate a Kubernetes Deployment

Deploy and run the Mocker worker on Kubernetes.

Simulate a Local Deployment

Run a frontend and Mocker worker from the command line.

Run a DynoSim Simulation

Run one workload through an offline simulated configuration with aisimulate predict --stack dynamo.

Mocker Engine Architecture

Scheduler, KV block manager, eviction, and timing internals.

Planner Configuration

The PlannerConfig field reference for the Dynamo Planner service.

Core and model

KV cache

Native G2 host offload

Scheduling

Timing

Data parallelism and workers

Performance modeling

AIS performance model

Disaggregation

Event and request transport

SGLang-specific

TensorRT-LLM-specific

Router advertisement

Environment variables

Related pages