> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Mocker CLI Reference `python3 -m dynamo.mocker` launches a simulated Dynamo worker that registers with the frontend, publishes KV events, and exercises the router and planner paths without a GPU. This page is the flag reference. For the deployment workflow and local/Kubernetes launch recipes, see [Simulate a Kubernetes Deployment](/dynamo/dev/knowledge-base/concepts/simulation/kubernetes-mocker) or [Simulate a Local Deployment](/dynamo/dev/knowledge-base/concepts/simulation/local-mocker); for the engine internals these flags configure, see [Mocker Engine Architecture](/dynamo/dev/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture). Run `python3 -m dynamo.mocker --help` for the complete option list supported by the installed version. AISimulate reuses the same engine core for offline virtual-clock prediction and recommendation, but its public YAML is separate from these live-worker flags. See the [DynoSim Replay CLI Reference](/dynamo/dev/reference/components/dyno-sim-replay-cli-reference) for the offline contract. The former public Replay online CLI remains unavailable, although the Python replay SDK retains online mode. ## Core and model **`--version`** `boolean` — default: false Print the Dynamo Mocker version and exit. --- **`--model-path`** `string` — default: null Hugging Face model ID or local path for the tokenizer. Set this for normal frontend requests. --- **`--model-name`** `string` — default: derived from --model-path Model name used in API responses. --- **`--endpoint`** `string` — default: auto-derived Dynamo endpoint string. Defaults are namespace-dependent, and prefill workers use a different default endpoint than aggregated or decode workers. --- **`--engine-type`** `string` — default: vllm Engine simulation type. Allowed values: vllm sglang trtllm --- **`--extra-engine-args`** `path` — default: null Path to a JSON file with canonical AISimulate launch configuration and optional `dynamo` runtime options. Engine fields belong under `engine`. This file overrides individual CLI arguments; see [Canonical Python Engine Configuration](/dynamo/dev/reference/components/dyno-sim-replay-cli-reference#canonical-python-engine-configuration). --- ## KV cache **`--num-gpu-blocks-override`** `integer` — default: auto Usable KV cache blocks per data-parallel rank. Non-AIS timing defaults to 16,384 blocks; AIS timing estimates capacity when this option is omitted. --- **`--block-size`** `integer` — default: engine-specific Tokens per KV cache block. Defaults are 64 for vLLM, 1 for SGLang, and 32 for TensorRT-LLM. --- **`--max-model-len`** `integer` — default: null Maximum sequence length, including prompt and generated tokens. Omitting the option leaves the simulated engine without a model-length limit. --- **`--enable-prefix-caching`** `boolean` — default: true Enable prefix caching. Pass `--no-enable-prefix-caching` to disable. --- **`--kv-cache-dtype`** `string` — default: auto KV cache dtype for the bytes-per-token computation. --- **`--kv-bytes-per-token`** `integer` — default: auto-computed KV cache bytes per token. Overrides the auto-computation. --- ## Native G2 host offload The vLLM mocker can simulate framework-native G1-to-host (G2) KV offload. Offloaded blocks are restored from host memory on a later prefix hit. As with the native vLLM offloading connector, G2 residency is published as `HostPinned` (CPU-medium) KV events alongside device events, so the KV router can credit a host-resident prefix with `host_cache_hit_weight` and route to the worker that holds it. Host offload requires `--engine-type vllm` and prefix caching. Host blocks are sized by `--kv-bytes-per-token`. G2 events are not published on the vLLM ZMQ wire (`--zmq-kv-events-ports`), which needs per-block token IDs that the simulated G2 does not carry; a router consuming that stream sees only G1. **`--num-host-blocks`** `integer` — default: null Enable native G2 host offload with this host cache capacity per data-parallel rank, in blocks. --- **`--host-offload-d2h-bandwidth-gbps`** `float` — default: AISimulate default Per-rank device-to-host bandwidth in decimal GB/s. `0` is unlimited. Requires `--num-host-blocks`. --- **`--host-offload-h2d-bandwidth-gbps`** `float` — default: AISimulate default Per-rank host-to-device bandwidth in decimal GB/s. `0` is unlimited. Requires `--num-host-blocks`. --- Offline replay can also simulate one G2 pool shared by every worker of the simulated deployment. Live mocker workers reject this scope, so pass it through the offline replay Python API: ```python from dynamo.replay import run_trace_replay engine_args = { "engine": { "backend": "vllm", "kv_cache_bytes_per_token": 131072, "native_host_offload": { "scope": "cluster_shared", "num_host_blocks": 65536, "kv_layout_id": "llama-3-8b-tp1", }, }, } report = run_trace_replay( "trace.jsonl", extra_engine_args=engine_args, num_workers=4, replay_mode="offline", router_mode="kv_router", ) ``` Offline replay does not compute bytes per token from a model, so set `engine.kv_cache_bytes_per_token` explicitly. `engine.native_host_offload` also accepts `latency_to_first_byte_ms` and the shared-pool bandwidth caps. G2 host offload does not support MTP (`--ais-nextn`). ## Scheduling **`--max-num-seqs`** `integer` — default: 256 Maximum concurrent sequences. --- **`--max-num-batched-tokens`** `integer` — default: 8192 Maximum tokens per batch. --- **`--enable-chunked-prefill`** `boolean` — default: true Enable chunked prefill. Pass `--no-enable-chunked-prefill` to disable. --- **`--preemption-mode`** `string` — default: lifo Decode eviction policy under memory pressure. `lifo` is vLLM v1 style. Allowed values: lifo fifo --- ## Timing **`--speedup-ratio`** `float` — default: 1.0 Timing speedup factor applied to simulated prefill and decode durations. --- **`--decode-speedup-ratio`** `float` — default: 1.0 Decode-only speedup multiplier, for example for Eagle speculation. --- **`--startup-time`** `float` — default: null Simulated startup delay, in seconds. --- ## Data parallelism and workers **`--data-parallel-size`** `integer` — default: 1 Number of DP replicas. --- **`--num-workers`** `integer` — default: 1 Workers per process. Prefer this over launching many separate mocker processes: all workers share one tokio runtime and thread pool. --- **`--stagger-delay`** `float` — default: 0 Delay between worker launches, in seconds. `0` disables staggering (default); `-1` enables auto mode. --- ## Performance modeling **`--planner-profile-data`** `path` — default: null Path to either a mocker-format `.npz` file or a profiler results directory. --- **`--reasoning`** `JSON string` — default: null JSON config for emitting reasoning token spans, with `start_thinking_token_id`, `end_thinking_token_id`, and `thinking_ratio`. --- **`--response-replay-trace-path`** `path` — default: null Mooncake JSONL trace whose `output_token_ids` provide response replay annotations. --- ## AIS performance model **`--ais-perf-config`** `JSON object or JSON/YAML file` — default: null Complete AISimulate `ForwardPassPerfModelConfig`, including immutable worker role, data roots, selection policy and nested estimator controls. Defaults to `estimation_mode: auto` and `fallback_policy: deny`. Do not combine this input with flat AIS identity flags. Cold regression is rejected by Router and Mocker. Mocker and Dynamo Replay require `pp: 1`; pipeline-parallel configurations are rejected. --- Legacy `--aic-*` spellings remain input aliases. Use `--ais-*` in new configurations. Opt-in flags for the AISimulate (AIS) latency model. Live Mocker workers do not use AIS by default. For the timing-model design, see [Mocker Engine Architecture](/dynamo/dev/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture#performance-model). **`--ais-perf-model`** `boolean` — default: false Use the canonical AISimulate performance-model API for latency prediction instead of the interpolated or polynomial models. Install `aisimulate` and use a supported `system/backend/version` tuple. --- **`--ais-system`** `string` — default: h200\_sxm when AIS is enabled AIS system name used with `--ais-perf-model`. The raw CLI value is unset when omitted; runtime resolution uses `h200_sxm` when AIS modeling is enabled. --- **`--ais-backend`** `string` — default: --engine-type Backend used for AIS performance-data lookup. Set it only to model timing for a different backend than the simulated scheduler. Allowed values: vllm sglang trtllm --- **`--ais-backend-version`** `string` — default: auto AIS performance-database version. Use `current`, `previous`, or `next` when the slot is available for the selected system and backend, or specify a version assigned to one of those slots. When unset, uses the release database's `current` slot. --- **`--ais-tp-size`** `integer` — default: 1 Tensor-parallel size for AIS latency prediction. Affects only AIS performance-model lookups, not mocker scheduling. --- **`--ais-moe-tp-size`** `integer` — default: null Mixture-of-Experts tensor-parallel size for AIS latency prediction. Required by some MoE models. --- **`--ais-moe-ep-size`** `integer` — default: null Mixture-of-Experts expert-parallel size for AIS latency prediction. Required by some MoE models. --- **`--ais-attention-dp-size`** `integer` — default: null Attention data-parallel size for AIS latency prediction. Required by some MoE models. --- **`--gpu-memory-utilization`** `float` — default: 0.9 vLLM GPU memory fraction used for AIS KV-capacity estimation. --- **`--mem-fraction-static`** `float` — default: 0.88 SGLang static memory fraction used for AIS KV-capacity estimation. --- **`--free-gpu-memory-fraction`** `float` — default: 0.9 TensorRT-LLM fraction of post-model-load free GPU memory used for AIS KV-capacity estimation. --- **`--ais-nextn`** `integer` — default: null Experimental Multi-Token Prediction (MTP) draft length, from 1 through 5. --- **`--ais-nextn-accept-rates`** `string` — default: null Experimental comma-separated conditional MTP acceptance rates. --- **`--ais-mtp-seed`** `integer` — default: 42 Base random seed for Mocker MTP burst sampling. --- ## Disaggregation **`--disaggregation-mode`** `string` — default: agg Worker mode. Allowed values: agg prefill decode --- **`--is-prefill-worker`** `boolean` — default: false Deprecated alias for `--disaggregation-mode=prefill`. Do not combine it with `--disaggregation-mode` or `--is-decode-worker`. --- **`--is-decode-worker`** `boolean` — default: false Deprecated alias for `--disaggregation-mode=decode`. Do not combine it with `--disaggregation-mode` or `--is-prefill-worker`. --- **`--bootstrap-ports`** `string` — default: null Comma-separated rendezvous base ports, one per worker in disaggregated mode. --- **`--kv-transfer-bandwidth`** `float` — default: null KV cache transfer bandwidth in GB/s. Set to `0` to disable. --- **`--kv-transfer-timing-mode`** `string` — default: full\_prompt Prompt footprint charged to a coordinated prefill/decode transfer. Allowed values: full\_prompt destination\_missing --- ## Event and request transport **`--event-plane`** `string` — default: auto Event transport. When the environment does not set a value, Mocker uses ZMQ with file or memory discovery and NATS with etcd or Kubernetes discovery. Allowed values: nats zmq --- **`--request-plane`** `string` — default: env-driven (tcp) Request transport. Allowed values: nats tcp --- **`--discovery-backend`** `string` — default: env-driven (etcd) Discovery backend. Allowed values: kubernetes etcd file mem --- **`--zmq-kv-events-ports`** `string` — default: null Comma-separated ZMQ PUB base ports for KV event publishing, one per worker. --- **`--zmq-replay-ports`** `string` — default: null Comma-separated ZMQ ROUTER base ports for gap recovery, one per worker. --- ## SGLang-specific Apply only when `--engine-type sglang`. Set the radix-cache page size with `--block-size`. **`--sglang-schedule-policy`** `string` — default: fifo SGLang scheduling policy. `fifo` is the default; `lpm` is longest prefix match. The deprecated `fcfs` alias maps to `fifo` and prints a warning when used. Allowed values: fifo lpm --- **`--sglang-page-size`** `integer` — default: null Deprecated alias for `--block-size` when `--engine-type sglang`. Prints a warning when used; other backends ignore it. If both flags are supplied for SGLang, their values must match. --- > **Warning** > > Both aliases are deprecated in Dynamo 1.6.0 and will be removed in Dynamo 1.8.0, after two > releases. Use `--sglang-schedule-policy fifo` and `--block-size` instead. **`--sglang-max-prefill-tokens`** `integer` — default: 16384 SGLang max prefill-token budget per batch. --- **`--sglang-chunked-prefill-size`** `integer` — default: 8192 SGLang chunked-prefill chunk size. --- **`--sglang-clip-max-new-tokens`** `integer` — default: 4096 SGLang admission-budget cap for max new tokens. --- **`--sglang-schedule-conservativeness`** `float` — default: 1.0 SGLang schedule conservativeness factor. --- ## TensorRT-LLM-specific Apply only when `--engine-type trtllm`. **`--trtllm-capacity-scheduler-policy`** `string` — default: guaranteed\_no\_evict TensorRT-LLM capacity scheduler policy. The Mocker currently supports only `guaranteed_no_evict`. --- ## Router advertisement Mocker accepts the same `--router-*` worker-advertisement options as the real backend launchers. Use them to override Router behavior for this worker set. See [Router Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning) for the shared fields. **`--router-mode`** `string` — default: inherit frontend Advertise a Router configuration for this worker set. When omitted, the worker advertises no Router configuration and inherits the Frontend configuration. Allowed values: round-robin random power-of-two kv direct least-loaded device-aware-weighted --- ## Environment variables | Variable | Default | Description | | --------------------------- | ------- | -------------------------------------------------------------------------------------- | | `DYN_MOCKER_KV_CACHE_TRACE` | off | Set to `1` or `true` to log structured native KV-cache allocation and eviction traces. | ## Related pages #### [Simulate a Kubernetes Deployment](/dynamo/dev/knowledge-base/concepts/simulation/kubernetes-mocker) Deploy and run the Mocker worker on Kubernetes. #### [Simulate a Local Deployment](/dynamo/dev/knowledge-base/concepts/simulation/local-mocker) Run a frontend and Mocker worker from the command line. #### [Run a DynoSim Simulation](/dynamo/dev/knowledge-base/concepts/simulation/simulation-runs) Run one workload through an offline simulated configuration with `aisimulate predict --stack dynamo`. #### [Mocker Engine Architecture](/dynamo/dev/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture) Scheduler, KV block manager, eviction, and timing internals. #### [Planner Configuration](/dynamo/dev/reference/components/planner-configuration) The PlannerConfig field reference for the Dynamo Planner service.