Planner Live Retune

Live retune is the Kubernetes form of Periodic Retune. Every retune.interval_seconds, the Planner cuts the window of request traces that just ended, creates DynamoGraphDeploymentRequests (DGDR) v1beta2 that replay the window for a grid of KV router parameters, scores the candidates the Runs report on retune.objective, and keeps the current parameters unless a candidate beats them by more than retune.min_improvement_percentage. Outside advisory mode it then sets the winner on every frontend router with PATCH /router_policy.

Live retune searches the three router knobs a DGDR Request can express: overlap_score_credit, prefill_load_scale and temperature. Replicas stay fixed: the Planner runs in retune-only mode and never scales.

How it works

1. Record. Each frontend writes a request_end record per request to its own directory on a shared ReadWriteMany claim (<segments_dir>/<dgd>/<pod>/).
2. Cut a window. At L_k = window end + grace_seconds, the Planner merges the records of the window into one canonical trace file, <windows_dir>/<dgd>/<window_end_ms>-<selection>.jsonl.gz, in a subprocess bounded by cut_budget_seconds. <selection> is a digest of the window’s start, end and cutoff (window end + grace_seconds), so a Planner restarted with another interval_seconds or grace_seconds cuts its own file and creates its own Requests instead of reusing those of a window with the same end.
3. Search. The Planner splits the router grid into at most run.max_shards Cartesian shards of at most 100 points and creates one DGDR v1beta2 Request per shard. Each Request replays the window file on the pinned retune.dgdr.engine with the replica counts of the DGD spec. One more single-point Request replays a grid point again as a common-random-numbers probe: its score must match the grid’s within a relative error of 1e-9.
4. Decide. The Planner reads each Run’s Candidates, scores them on retune.objective, and decides with the same rules as replay retune. A decision is persisted in a Lease before it reaches any router.
5. Apply. The Planner sets all eight router policy parameters of every frontend router to the decided values, and keeps them there: a manual change to them is reverted.

Windows are aligned to retune.dgdr.phase_seconds + k × interval_seconds in epoch time, so a restarted Planner continues the same windows. Leadership and state live in two Leases owned by the DGD, <dgd>-planner-retune and <dgd>-planner-retune-state. The first window after retune is first enabled is skipped, since its trace is incomplete.

Every DGDR trace Run replays with common random numbers: ordinal request ids and a fixed router seed. A Run’s score therefore does not depend on the other candidates or the Run’s parallelism. This applies to all DGDR trace Runs, not only those the Planner creates.

Prerequisites

• Operator with DGDR v1beta2 traces. Install the platform chart with dynamo-operator.dgdrV1Beta2.enabled=true and an operator that accepts spec.workload.trace. Also set dynamo-operator.dgdrV1Beta2.plannerLiveRetune=true to grant the Planner ServiceAccount the access it needs (see RBAC). In a cluster-wide install, the operator must also hold the DGDR v1beta2 Run and Candidate permissions of ai-dynamo/dynamo#17308; see RBAC.
• Run Job scheduling. The namespace needs a LimitRange for the Run Jobs, and the operator must discover GPUs, through DCGM or GPU discovery with labeled nodes. See Auto Deploy with DGDR.
• A public model. The Run Job has no Hugging Face credentials and no model mount, so retune.dgdr.model must be a public Hugging Face repository id that AISimulate can resolve.
• A ReadWriteMany claim in the DGD namespace, dedicated to traces. Create segments/ and windows/ at its root, writable by the user the frontend and Planner containers run as. The Planner writes each window file with mode 0644, so the Run Job, which reads it as UID 1000 with supplemental group 65532, can read it whatever user the Planner runs as.
• A KV-router frontend with default routing fields, and workers that do not set their own router mode or pass a value the Planner cannot read where an option could stand. See Deployment rules.
• One Planner replica.

Configure the frontend

Add the trace environment and the claim to the frontend container of the DGD. The planner checks these values against retune.dgdr at startup and whenever the DGD changes.

spec:
  components:
  - name: Frontend
    type: frontend
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          command: [python3]
          args: [-m, dynamo.frontend, --router-mode, kv]
          env:
          - name: DYN_POD_NAME
            valueFrom: {fieldRef: {fieldPath: metadata.name}}
          - {name: DYN_REQUEST_TRACE, value: "1"}
          - {name: DYN_REQUEST_TRACE_RECORDS, value: request_end}
          - {name: DYN_REQUEST_TRACE_SINKS, value: file}
          - {name: DYN_REQUEST_TRACE_FILE_FORMAT, value: jsonl_gz}
          - name: DYN_REQUEST_TRACE_FILE_PATH
            value: /trace/segments/$(DYN_PARENT_DGD_K8S_NAME)/$(DYN_POD_NAME)/req
          - {name: DYN_REQUEST_TRACE_FILE_ROLL_BYTES, value: "67108864"}
          - {name: DYN_REQUEST_TRACE_CAPACITY, value: "8192"}
          volumeMounts:
          - {name: request-traces, mountPath: /trace}
        volumes:
        - name: request-traces
          persistentVolumeClaim: {claimName: request-traces}

• DYN_REQUEST_TRACE_FILE_PATH is a file prefix directly under <mount_path>/<segments_dir>/$(DYN_PARENT_DGD_K8S_NAME)/$(DYN_POD_NAME)/, such as req: the part after $(DYN_POD_NAME)/ must not contain /. <mount_path> is where the frontend mounts retune.dgdr.claim_name. The operator injects DYN_PARENT_DGD_K8S_NAME.
• Records are dropped without a count for prompt_embeds, multimodal, n > 1 and best_of > 1 requests, and when the writer lags or a batch write fails.
• The trace holds unkeyed xxh3 hash chains of the prompts. Anyone who can create pods in the namespace can mount the claim and read them.

See Agent Tracing for the request_end record.

Configure the Planner

Mount the same claim in the Planner container at retune.dgdr.mount_path, and pass a configuration like the one below with --config:

environment: kubernetes
backend: vllm
mode: agg
advisory: true                     # start advisory; see Roll out
enable_load_scaling: false         # retune-only: both flags false
enable_throughput_scaling: false
retune:
  interval_seconds: 21600          # T: window length and decision cadence
  grace_seconds: 120               # covers p99 request time + 1 s flush + claim visibility
  timeout_seconds: 20400           # required; at most T - 600; includes the window cut
  decision_latency_seconds: 0
  objective: GoodputPerGPU
  sla_ttft_ms: 500
  sla_itl_ms: 30
  router: default                  # 3 x 9 x 4 = 108 points
  executor: dgdr
  dgdr:
    phase_seconds: 0
    claim_name: request-traces
    mount_path: /trace
    segments_dir: segments
    windows_dir: windows
    model: Qwen/Qwen3-32B
    sku: h200_sxm
    engine:
      mode: aggregated
      backend: vllm
      context_length: 32768
      workers:
        aggregated:
          parallelism:
            preset:
            - {tensor: 2, pipeline: 1, attention_data: 1, moe_tensor: 1, moe_expert: 1}
          scheduler: {max_batched_tokens: 8192, max_sequences: 256}
        prefill: null
        decode: null
    cut_budget_seconds: 900
    job_startup_seconds: 300
    run:
      parallelism: 8
      candidate_timeout_seconds: 1800
      max_shards: 3
    keep_requests: 5
    max_window_rows: 5000000
    promote_last_advisory: false

retune.dgdr fields:

For the other retune fields and the limits that executor: dgdr adds to them, see the Retune field reference.

Live retune takes its scoring SLA when it starts: retune.sla_ttft_ms and retune.sla_itl_ms, or the Planner’s ttft_ms and itl_ms where they are unset. An update of ttft_ms or itl_ms through the runtime reconfiguration API that would change that SLA fails with HTTP 422. To change the SLA, restart the Planner with the new values; to keep ttft_ms and itl_ms changeable at runtime, set both retune SLA fields.

Engine section

retune.dgdr.engine becomes spec.aisim.inputOverrides.engine of every Request, so every value in it must be a scalar. A {choices: ...} or {range: ...} domain, or an omitted knob that AISimulate searches by default, would make the Run search engine settings as well.

• mode is aggregated for mode: agg and disaggregated for mode: disagg.
• Each worker role of the mode (aggregated, or prefill and decode) sets parallelism.preset to a list of exactly one entry with tensor, pipeline, attention_data, moe_tensor and moe_expert, and sets scheduler.max_batched_tokens and scheduler.max_sequences.
• The preset entry may also set prefill_context, but only to 1, and must not set decode_context: the pinned AISimulate recommendation does not run context parallelism.
• Do not set replicas anywhere: the Planner fills them in from the DGD spec at each launch and sets spec.hardware.gpu.budget to their GPU total.
• Write roles the mode does not use as null, or leave them out.
• Set the model and GPU with retune.dgdr.model and retune.dgdr.sku, not in the engine.
• Leave backend_version out to let the pinned AISimulate pick its version for retune.dgdr.sku. If you set it, it must be a version AISimulate accepts for that GPU; otherwise every Run fails with UnlistedQueryVersionError.

Time budget

Each shard must finish before the window’s deadline, L_k + timeout_seconds. A shard of n points needs

need(n) = cut_budget_seconds + job_startup_seconds
          + (1 + ceil((n - 1) / run.parallelism)) x run.candidate_timeout_seconds

AISimulate evaluates the first candidate alone, hence the 1 +. The Planner rejects a configuration whose largest grid shard needs more than timeout_seconds, because the baseline can fall in any shard and a window without a baseline score fails. With the defaults, the 108-point grid splits into 3 shards of 36 points and needs 900 + 300 + 6 × 1800 = 12000 seconds. Measure replay time and memory on your trace before relying on these defaults.

Deployment rules

The Planner checks three kinds of rules. None of them stops a running Planner except a configuration error at startup.

Configuration, checked when the Planner starts:

Deployment, checked at startup, when the DGD changes and before every router round. While one fails, the Planner launches no windows, resets the routers to their startup parameters, and retries:

Health, checked every round. While one fails, windows do not launch; nothing is reset:

The Planner only warns when AISimulate does not list the model among its default models (P6).

Roll out

1. Prepare the claim and its directories, then add the frontend trace configuration and the Planner mount.

2. Start with advisory: true. In advisory mode the Planner records the decision it would make and changes no router value it has not set itself. Watch dynamo_planner_retune_ready and the decision history:

   kubectl get lease <dgd>-planner-retune-state -n <namespace> \
     -o jsonpath='{.metadata.annotations.nvidia\.com/planner-retune-history}'

3. Set advisory: false. With retune.dgdr.promote_last_advisory: true, the Planner applies the last advisory recommendation right away if it is at most one interval old and the configuration has not changed since.

Disable

1. Set advisory: true and wait until dynamo_planner_retune_router_converged is 1 and dynamo_planner_retune_rollback_pending is 0. The routers are then back at their startup parameters. The mode stored in the state alone does not show that.

2. Remove retune from the Planner configuration.

3. Remove the trace environment from the frontend.

4. Delete the Planner’s Requests and both Leases. Select the Requests by the DGD’s metadata.uid: the nvidia.com/planner-retune-dgd label holds a hash instead of the name when the name is longer than 63 characters.

   kubectl delete dynamographdeploymentrequests.nvidia.com -n <namespace> \
     -l nvidia.com/planner-retune-dgd-uid=$(kubectl get dgd <dgd> -n <namespace> -o jsonpath='{.metadata.uid}')
   kubectl delete lease -n <namespace> <dgd>-planner-retune <dgd>-planner-retune-state

   While the chart still grants the live retune RBAC, a Planner without retune does this on its own: it looks for the Leases in the background after it starts (once more after a transient API error), resets any router the removed retune changed, then deletes the Requests and both Leases.

5. Delete <segments_dir>/<dgd> and <windows_dir>/<dgd> from the claim.

Before upgrading the operator or its CRDs, switch to advisory or disable retune, then delete the retune Requests.

Security

• Frontend admin API. /router_policy is served on the frontend HTTP port, next to the OpenAI API, without authentication, and is on by default. An Ingress that forwards / exposes it. Deny the admin paths at L7 and allow in-cluster access only from the ingress controller and the Planner. Do not set DYN_DISABLE_FRONTEND_ADMIN_API on the frontend: the Planner reports a frontend whose /router_policy answers 404 as not ready.
• Ownership. Outside advisory mode the Planner owns the eight router policy parameters and reverts manual changes.
• Trace data. See the privacy note in Configure the frontend.

RBAC

With dynamo-operator.dgdrV1Beta2.enabled=true and dynamo-operator.dgdrV1Beta2.plannerLiveRetune=true, the platform chart adds these rules to the Planner Role (namespace-restricted installs) and ClusterRole (cluster-wide installs). Both values default to false; dgdrV1Beta2.enabled alone grants the Planner nothing new.

The chart has one Planner role for the whole install, so Planners without retune get these rules too. The Planner also lists pods with its existing permission. These rules are namespace-wide: a Role or a RoleBinding of the ClusterRole has no resourceNames, because the Planner creates its Leases and Requests under per-DGD names. As a result, any workload that runs as the Planner ServiceAccount can:

• create DGDR Requests, which start Run Jobs, in its namespace;
• read, update or delete any Lease in its namespace, not only the two retune Leases. In a namespace-restricted install, the Planner Role lives in the release namespace, which by default also holds the operator’s leader-election Lease, so such a workload can disrupt operator leader election.

In a cluster-wide install, the operator binds the Planner ClusterRole into each DGD namespace with a RoleBinding, and Kubernetes lets it bind only permissions it holds itself. Set dynamo-operator.dgdrV1Beta2.plannerLiveRetune=true there only with an operator whose own role grants get on dynamographdeploymentruns and list on dynamographdeploymentcandidates, which the DGDR v1beta2 operator change in ai-dynamo/dynamo#17308 adds. Without it, the API server rejects the Planner RoleBinding in every namespace that does not have one yet, and the DGDs there fail to reconcile their Planner RBAC, with or without live retune.

Metrics

The Planner exports these on metric_reporting_prometheus_port:

Limitations