Planner Guide

The Dynamo Planner is an autoscaling controller that adjusts prefill and decode engine replica counts at runtime to meet latency SLAs. It reads traffic signals (Prometheus metrics or load predictor output) and engine performance models to decide when to scale up or down.

Forward Pass Metrics (FPM) are per-iteration scheduler records from inference workers. They describe batch composition, queue depth, token counts, and forward-pass duration. The Planner uses these records to tune its performance model from live traffic or to build a regression model when a native AISimulate estimate is unavailable.

For a quick overview, see the Planner overview. For architecture internals, see Planner Design.

Scaling Modes

The planner supports four optimization targets that determine how scaling decisions are made:

• throughput (default): Uses static thresholds on queue depth and KV cache utilization. No SLA targets or profiling needed. Works out of the box.
• latency: Same approach as throughput but with more aggressive thresholds — scales up earlier and tolerates less queuing. Ideal for latency-sensitive workloads.
• load: Uses user-defined prefill queue token thresholds and decode KV utilization thresholds for reactive load-based scaling.
• sla: Uses the Planner engine-query layer with forward-pass estimates from the AIConfigurator compatibility API in the aisimulate wheel, plus online FPM tuning or FPM regression fallback, to target specific TTFT/ITL values. Supports both throughput-based (predictive) and load-based (reactive) scaling modes. For advanced users who need precise SLA control.

When to use which:

• Start with throughput (the default) — it works immediately with no configuration.
• Switch to latency if your workload has strict latency requirements and you prefer to over-provision rather than queue.
• Use load when you want direct control through prefill queue and decode KV utilization thresholds.
• Use sla when you want to target specific TTFT/ITL values with native AISimulate estimates, optional bootstrap profiling data, or live FPM warmup.

PlannerConfig Reference

The planner is configured via a PlannerConfig JSON/YAML object. When using the profiler, place this under spec.features.planner in the DGDR spec. Any PlannerConfig field listed below can be set there; DGDR passes that object to the Planner service for validation.

spec:
  features:
    planner:
      mode: disagg
      backend: vllm
      # optimization_target defaults to "throughput" — works out of the box

For SLA-based scaling:

spec:
  features:
    planner:
      optimization_target: sla
      enable_throughput_scaling: true
      enable_load_scaling: false
      pre_deployment_sweeping_mode: rapid
      mode: disagg
      backend: vllm

To evaluate Planner behavior without changing replica counts, turn on advisory mode:

spec:
  features:
    planner:
      advisory: true

Advisory mode is suggestion-only. The Planner computes recommended replica counts, logs them, exports them as diagnostics, and shows them in HTML reports. The recommendations are not applied as scaling decisions: the Planner does not execute scaling actions or change the deployment. Planner-in-the-loop replay honors it too: the replay keeps the simulated deployment fixed, reports the recommendations, and records retune decisions, which run only in simulation, without applying them; see Replay in Advisory Mode.

Optimization Target

When optimization_target is throughput, latency, or load, load-based scaling is automatically enabled and throughput-based scaling is disabled. The ttft_ms/itl_ms fields are ignored.

Scaling Mode Fields (SLA mode)

At least one scaling mode must be enabled when using optimization_target: sla.

Pre-Deployment Sweeping

SLA mode constructs each role’s model through aisimulate_core.sdk.RustForwardPassPerfModel.best_available(config). Set ais_perf_model.roles to the complete AISimulate configuration for each deployed role: prefill and decode for disaggregated deployments, or aggregated for an aggregated deployment. Each role key must match its worker_type.

New configurations default to estimation_mode: auto and fallback_policy: deny. Auto searches op_level, fpm_interpolation, then fpm_regression; deny prevents fallback only when you explicitly select a mode. Without ais_perf_model, the Planner creates a cold fpm_regression model and trains it from self-benchmark or live forward-pass metrics (FPM).

Each role accepts the complete upstream schema, including ordered systems_paths, database_mode, transfer_policy, quantization and speculation settings, and nested estimator_config controls. Explicit estimator controls override the Planner’s sampling defaults. fpm_sample_bucket_size: 16 corresponds to a [4, 4] grid; regression and correction may use separate grids. Unknown fields and conflicting worker identities are rejected.

Configure model identity and estimator controls through ais_perf_model.roles. The retired aic_perf_model field and hf_id / parallel-pick shape are rejected.

At startup, the planner always tries to fetch self-benchmark results from the get_perf_metrics Dynamo endpoint. If unavailable, it falls back to rapid-mode AIC interpolation data or profiler-generated data (npz or JSON) at profile_results_dir when configured. These sources are converted to ForwardPassMetrics and used to tune or bootstrap the perf model. With pre_deployment_sweeping_mode: none, the planner can still start; throughput decisions report model_not_ready until native AIS estimates are available or enough live FPMs have warmed the regression fallback.

Configure the role models in a DynamoGraphDeploymentRequest:

spec:
  features:
    planner:
      optimization_target: sla
      mode: disagg
      ais_perf_model:
        roles:
          prefill:
            model: Qwen/Qwen3-32B
            system: h200_sxm
            backend: vllm
            worker_type: prefill
            estimation_mode: auto
            fallback_policy: deny
            tp: 1
          decode:
            model: Qwen/Qwen3-32B
            system: h200_sxm
            backend: vllm
            worker_type: decode
            estimation_mode: auto
            fallback_policy: deny
            tp: 1
            estimator_config:
              fpm_regression:
                sampling:
                  bins_per_axis: [4, 4]
                  max_observations: 64
                min_observations: 5

Throughput-Based Scaling Settings

Load-Based Scaling Settings

General Settings

Traffic Prediction Settings

KV hit rate and speculative decode accept length are runtime engine/router signals, not traffic shape. The planner stores the latest valid observation for each signal and reuses it until a newer valid value arrives. On cold start, missing KV hit rate means no prefix-cache discount, and missing accept length means 1.0.

Kalman Filter Settings

Diagnostics Reports

Runtime Minimum Endpoint API

The Planner listens on 127.0.0.1:<control_api_port> and supports GET and partial PATCH requests at /v1/min-endpoints. The API has no authentication and is not exposed by a Kubernetes Service. It uses prefill_min_endpoint and decode_min_endpoint in disaggregated mode, the active component’s field in single-component mode, and min_endpoint in aggregated mode. Every mode also accepts min_gpu_budget and max_gpu_budget. A response includes the active endpoint fields and both GPU budgets. Updates are process-local, are not written back to the Planner ConfigMap, and apply to the next planner tick.

In Kubernetes, port-forward to the Planner pod and patch the active mode’s field. The following example updates a Planner in disagg mode:

kubectl port-forward pod/<planner-pod> 9086:9086
curl http://127.0.0.1:9086/v1/min-endpoints
curl --request PATCH http://127.0.0.1:9086/v1/min-endpoints \
  --header 'Content-Type: application/json' \
  --data '{"prefill_min_endpoint": 8, "decode_min_endpoint": 8, "min_gpu_budget": 16, "max_gpu_budget": 64}'

The update is atomic. Use -1 to disable either GPU budget. The Planner rejects malformed values, fields that are inactive for the current mode, a min_gpu_budget greater than max_gpu_budget when both are enabled, and minimum footprints that exceed max_gpu_budget or the configured power budget. The same checks run at startup, so an infeasible minimum configuration fails before the Planner enters its tick loop. Scale-up has no per-component maximum endpoint setting; the existing GPU, power, Global Planner, and cluster-capacity limits remain the upper bounds.

The same diagnostic signals surfaced in these reports are also exported as Prometheus metrics under the dynamo_planner_* prefix—for example estimated TTFT/ITL (dynamo_planner_estimated_ttft_ms, dynamo_planner_estimated_itl_ms), recommended replica counts (dynamo_planner_predicted_num_prefill_replicas, dynamo_planner_predicted_num_decode_replicas), per-engine capacity and FPM queue depths, and load/throughput scaling decision enums.

The Replica Counts plot overlays actual prefill/decode replicas with discrete recommendation markers for the Planner’s recommended prefill/decode replicas. When advisory: true, these recommended counts are suggestions only; the Planner records what it would do without applying the change.

Unavailable Scaling Inputs

dynamo_planner_signal_unavailable{component,signal} reports missing inputs on their most recent scheduled observation. After the consecutive-observation threshold defined by SIGNAL_DEGRADATION_OBSERVATIONS in planner_metrics.py is reached, the Planner logs Planner degraded signal and increments dynamo_planner_degraded_signals_total{component,signal} once for that episode. A successful observation clears the gauge and allows a later outage to produce a new alert. Ticks that do not request a signal do not count as missing samples.

The signals distinguish missing FPM, stale FPM, worker discovery join failures, missing speculative accept length, and unavailable traffic observations. load_inputs and throughput_inputs report missing_engine_metadata from the scaling paths’ own metadata checks, such as missing KV capacity or batching limits. Normal performance-model warmup (insufficient_data or model_not_ready) remains visible in decision diagnostics and does not trigger a degraded-input alert. For a pool with ready workers, metadata health changes only when a scaling path actually evaluates that metadata; skipped, idle, or zero-demand decisions do not clear an existing metadata alarm. An explicitly empty pool clears its metadata alarm. Runtime FPM samples expire after three load-adjustment intervals measured from local receipt. Idle heartbeats refresh that receipt time. Samples from workers absent from discovery are excluded and logged with their worker IDs. Each discovered worker must provide a fresh sample for every DP rank declared in its model card. A worker or rank that has never published is unavailable; the Planner does not use a partial component snapshot to infer idle capacity. A snapshot is also unavailable if any rank’s payload is malformed or uses an unsupported FPM schema version.

Missing, non-finite, or failed router request-counter queries produce unavailable demand, matching the frontend collection behavior. An explicit counter value of zero remains a valid idle observation.

Fresh idle FPM can reduce an SLA pool without waiting for a cold performance model. A valid zero-demand throughput prediction can also reduce a pool without a capacity estimate. Missing observations do not count as idle. Replica floors, GPU budgets, pending rollout checks, and downstream actuation constraints still apply. A restarted Planner uses observed deployment counts to resume decisions. These model-independent idle paths retain at least one worker per active role, even when min_endpoint: 0: without a publisher or a ready capacity model, they cannot independently wake a fully stopped pool.

Scheduling / plugin pipeline

The planner runs through the builtin plugin pipeline by default. The base pipeline cadence lives under the scheduling sub-tree of PlannerConfig; plugin registration, transport, and auth settings live under plugin_registration.

Existing planner fields still drive the builtin plugins:

• load_adjustment_interval_seconds schedules builtin_load_propose, which reads FPM and worker-count observations and applies the current load-based algorithm.
• throughput_adjustment_interval_seconds schedules builtin_load_predict and builtin_throughput_propose. The throughput proposer requires the prediction from the same tick, so it only fires when the predict plugin fires.
• When both builtins propose targets in the same tick, load-based scaling runs after throughput-based scaling and preserves the existing behavior: throughput updates the lower-bound replicas, then load-based scaling can adjust above that floor and apply the global GPU budget clamp.
• After the plugin pipeline finishes, the planner applies the same final effective component minimums and GPU-budget safety checks to built-in and external-plugin targets before scaling the deployment.

DGDR example

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-deployment
spec:
  model: Qwen/Qwen3-0.6B
  features:
    planner:
      optimization_target: sla
      enable_load_scaling: true
      ttft: 200.0
      itl: 10.0
      pre_deployment_sweeping_mode: rapid
      scheduling:
        tick_max_duration_seconds: 30.0
      plugin_registration:
        transport:
          request_timeout_seconds: 5.0

Periodic Retune (Experimental)

Retune lets the Planner tune its own runtime parameters from the traffic it served. Every retune.interval_seconds, it takes the window of the request trace that just ended and replays it once per candidate parameter point, on the deployment’s engine configuration and from the replica counts at the window start. It scores each candidate on retune.objective, by default SLA goodput per GPU: SLA-good output tokens per second of the window, divided by the GPUs provisioned. The objective takes the DGDR v1beta2 spec.objective.metric values, such as ThroughputPerGPU or TTFT; see Objectives. When the best candidate beats the current point on the same window by more than min_improvement_percentage, the Planner switches to it.

Retune searches parameters a running deployment can change without a restart:

• Router parameters of the KV router, in retune.router: overlap_score_credit, prefill_load_scale, temperature, and four more /router_policy parameters.
• Planner runtime fields, in retune.planner: load-scaling sensitivity, the load predictor, minimum worker counts, GPU budgets, max_throughput_scaling_replicas, and the thresholds of optimization_target: load.

The engine, the topology, the router mode, and Planner settings fixed at startup, such as the scaling intervals and ttft_ms / itl_ms, stay as configured.

Retune supports two scenarios:

With advisory: true, the Planner still runs every sweep but only records the decision it would apply, with outcome advisory; the router and Planner parameters keep their starting values.

A sweep takes time, so the decision lands at least one Planner tick after the sweep launches, and later when decision_latency_seconds models the sweep’s duration. The old parameters serve traffic until then. Each candidate is replayed once, and all candidates of a window share one random seed, so a candidate that changes no routing or scaling decision ties the current point and is kept; see Candidates and the Sweep. A candidate that changes decisions is judged on one replay of one window, so min_improvement_percentage sets the margin it must clear; see Measure the Gain.

This minimal fixed-replica configuration searches the 108-point default router domain every six hours. It is a PlannerConfig for the planner_config argument of dynamo.replay.run_trace_replay (see Run Retune from Python), not a DGDR spec.features.planner fragment: a live Planner rejects it. In aisimulate predict, the planner section takes retune and the two scaling flags, and mode comes from engine.mode; see Scenario 1.

mode: disagg
enable_load_scaling: false
enable_throughput_scaling: false
retune:
  interval_seconds: 21600
  router: default

To run retune and read its decisions, see Simulate Retune. For windows, candidates, scoring, and decision timing, see Planner Design. For every field and the search-domain syntax, see the retune reference.

Integration with Profiler

When the profiler runs with planner enabled, it:

1. Selects the best prefill and decode engine configurations
2. Generates engine performance data (prefill TTFT vs ISL, decode ITL vs KV-cache utilization)
3. Saves the PlannerConfig and performance data into separate Kubernetes ConfigMaps
4. Adds the planner service to the generated DGD, configured to read from those ConfigMaps

The planner receives its config via --config /path/to/planner_config.json which is mounted from the planner-config-XXXX ConfigMap. When thorough bootstrap data is generated, profiling data is mounted from the planner-profile-data-XXXX ConfigMap.

See the Profiler Guide for the full profiling workflow and how to configure pre-deployment sweeping.

Hierarchical Deployments

If you want one public endpoint for a model but multiple private DGDs optimized for different request classes, use a hierarchical deployment:

• one control DGD with Frontend, GlobalRouter, and GlobalPlanner
• one or more prefill pool DGDs
• one or more decode pool DGDs

In the current workflow, run profiling independently for each intended pool, then compose the final control DGD plus pool DGDs manually. See the Global Planner Guide.

See Also

• Planner overview — Why LLM inference needs a different autoscaler
• Planner Design — Architecture and algorithm internals
• Planner Examples — Planner-specific configuration examples
• Simulate Planner Decisions with AISimulate — Replay a trace through the Planner offline, including retune
• DGDR Templates — DGDR YAML examples, sample configurations, advanced patterns
• Global Planner Guide — Multi-DGD coordination, shared GPU budgets, single-endpoint multi-pool deployments
• Profiler Guide — How profiling data is generated