DynamoGraphDeployment (DGD) Reference

A DynamoGraphDeployment (DGD) is the canonical description of a running Dynamo inference graph. You list the components that make up the graph — frontend, workers, prefill and decode workers, planner, EPP — and the operator reconciles them into DynamoComponentDeployment resources, pods, and Services.

A DGD is the resource that persists and serves traffic. Author one directly when you have a known-good configuration or a tuned recipe; generate one from intent with a DynamoGraphDeploymentRequest (DGDR).

This page documents the nvidia.com/v1beta1 API — the served, current version.

LPX components are experimental and require the operator’s lpx.enabled setting. When disabled, admission rejects new or changed LPX components but permits otherwise valid updates that leave them unchanged, and deletion. Existing LPX DGDs report a disabled status without further workload reconciliation; installed CRDs and workloads are preserved.

Minimal example

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: my-graph
spec:
  backendFramework: vllm
  components:
    - name: frontend
      type: frontend
      replicas: 1
      podTemplate:
        spec:
          containers:
            - name: main
              image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.0
    - name: worker
      type: worker
      replicas: 1
      podTemplate:
        spec:
          containers:
            - name: main
              image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0

Grove update strategy

Set metadata.annotations["nvidia.com/grove-update-strategy"] on the DGD to select the generated Grove PodCliqueSet update strategy. Accepted values are Coherent, RollingRecreate, and OnDelete. This annotation belongs to the DGD’s metadata; spec.annotations propagates annotations to child resources instead.

Grove uses RollingRecreate unless this annotation explicitly selects another strategy. This applies to aggregated, disaggregated, and LPX deployments, regardless of operator origin version. Neither the deprecated component minAvailable nor a native minimum in providerOverride enables Coherent.

To opt into Coherent, add:

metadata:
  annotations:
    nvidia.com/grove-update-strategy: Coherent

Starting with Dynamo 1.6.0, new Grove-backed DGDs that omit all deprecated component minima default provider-native minAvailable to 1. Existing DGDs retain their persisted legacy fields. This availability defaulting is independent of the rollout strategy: both forms use RollingRecreate without an annotation. Mixing the old and native minimum forms anywhere in the DGD is rejected; migrate all components in one update. An explicit strategy annotation applies to both PodCliqueSets in a hybrid graph.

Use Grove v0.1.0-alpha.15 or later and its matching CRDs for Coherent updates. Upgrade externally managed Grove controllers and CRDs together. The operator writes the requested strategy and reports API validation or admission rejections through its normal failure condition and reconciliation retry path; it does not substitute a different strategy.

Grove v0.1.0-alpha.15 fixes corrective Coherent updates that stall when old replicas are unavailable (#873). For older Grove installations, review the recovery limitation before opting in. Changing or removing the strategy annotation cannot unblock a stalled active update. See Recover from a stalled Coherent update for the DGD recreation procedure and the limitations of pod deletion workarounds.

Coherent coordinates compatible replacement capacity within one PodCliqueSet (PCS). Grove selects components whose rendered pod templates change and gang-schedules a minimum viable unit of their replacements. Separate PCSes, including the ordinary and LPX PCSes in a hybrid graph, roll independently.

Dynamo includes a shared worker-generation hash in rendered worker templates. A pod-template edit to one worker can therefore roll every worker component, including workers whose authored pod template did not change. Non-worker components such as the frontend participate when their own rendered template changes. Coherent does not enforce traffic isolation between versions; Dynamo’s discovery and routing must keep requests within compatible worker generations.

Coherent capacity and disruption

Grove has no surge support: it removes old capacity before replacements become available. A larger minAvailable therefore requires a larger disruption budget; it is not a promise that this many replicas will remain serving throughout the update. Increasing maxUnavailable permits more unavailable capacity without reducing the MVU size. The budget gates further rollout deletions; unrelated failures can independently reduce availability.

For a component with eight replicas:

Grove can roll replicas beyond the minimum unit through additional tail steps. Progress depends on scheduling and remaining availability budget; it does not always wait for every replica in the previous batch to become Ready. Coherent does not guarantee an exact P/D ratio at every instant or zero downtime. For the detailed step plan, see Grove’s coherent update design.

For new DGDs, use the provider-native minAvailable: 1 default unless the application requires a larger viable unit. Keep the disruption budget at its default until the remaining capacity has been validated under representative traffic. Size spare serving capacity before a rollout; adding a larger budget does not add surge capacity. minAvailable is immutable after DGD creation, so review it before creating a deployment or opting an existing deployment into Coherent.

Grove rejects replica changes throughout an active Coherent rollout. Before changing replicas, the Dynamo operator synchronizes the desired PodCliqueSet configuration and, after a write, waits for its cache to observe that configuration. For Coherent, it also waits until Grove acknowledges the current configuration (status.observedGeneration matches metadata.generation) and any active rollout completes. Missing progress or completed progress from an older generation does not acknowledge a new configuration. Initial configuration acknowledgement does not require pods to be Ready or LPX scheduler requests to exist. During this wait the operator continues observing workload readiness and component status and reports ScalingDeferred when a replica change is pending. PodCliqueSet watch events resume scaling. RollingRecreate rollouts do not block replica changes once the configuration is observed, even if Grove’s observed generation lags. Unexpected API or admission failures use the normal reconciliation retry path and report Failed with Ready=False while the failure persists. A rejected scale write caused by lag between the Grove and Dynamo caches can therefore briefly report a failure until a retry succeeds. External scalers writing directly to Grove’s scale subresources still receive the admission rejection; do not rely on autoscaling to supply capacity during an active Coherent rollout.

For LPX workloads with explicit replica counts, scheduling deadline cleanup also waits until capacity can be reduced before deleting expired pipeline requests. Workloads with omitted replica counts retain external capacity ownership, and their expired requests can still be cleaned up during a rollout.

For RollingRecreate, maxUnavailable defaults to 1; rolling update configuration is rejected with OnDelete. See Grove’s defaulting implementation.

Set maxUnavailable through a component-level provider override; see Grove rollout budget overrides. Without an override, Dynamo leaves it unset and Grove supplies the defaults above.

Configure native minimum availability

Set an explicit minimum at component scope. Dynamo resolves the target when omitted, so generated PCS names are not required. An omitted minimum resolves to 1 during rendering; admission leaves an absent providerOverride absent and does not add availability fields to a topology-only override.

A standalone prefill component can specify:

providerOverride:
  apiVersion: grove.io/v1alpha1
  value:
    spec:
      minAvailable: 1

For a scaling-group component, use value.minAvailable: 1 instead. A shared LPX draft has no independent availability override; configure its target component. Root and multinode role overrides remain topology-only. LPX topology overrides remain unsupported.

The effective minimum must be a positive integer and fit a positive replica count; ordinary Grove components may scale to zero without clearing their minimum. With omitted LPX replicas, the minimum seeds the initial scaling-group capacity.

Admission returns a deprecation warning when a component introduces or changes the old minAvailable field, independently of the rollout strategy. Updates that leave the legacy value unchanged, including replica-only and metadata edits, do not repeat the warning. To migrate an existing DGD, remove every deprecated component minAvailable and put the same values in the corresponding component provider overrides in one update, including defaulted values on frontends and other components. Admission enforces effective-value immutability across both forms. Migration leaves the rollout strategy unchanged; select Coherent separately with the annotation. Moving the same values leaves worker hashes and rendered pod templates unchanged; subsequent workload updates use the selected strategy. Review the disruption implications before opting in.

A shared LPX draft has a legacy minimum of 1 but no independent scaling group. Remove that legacy field without adding a draft override; its effective default remains 1, and the target component owns the native workload minimum.

Both fields retain their native hub/spoke conversion mapping. Converting between v1alpha1 and v1beta1 preserves the chosen form and does not migrate a deployment or select a strategy.

Spec reference

Status

The operator maintains observed state under status.

Additional Types

Nested struct types referenced by the fields above, broken out here to keep the field lists shallow. Types prefixed core/v1. or metav1. are standard Kubernetes types and link to their Go package documentation instead of being expanded.

ProviderOverride

A sparse provider-native fragment, supported only for Grove-backed DGD resources. The same shape is used at graph, component, and role scope; the location determines the allowed target. Standalone DCDs do not support provider overrides.

Grove rollout budget overrides

Requires Grove v0.1.0-alpha.15 or later, which fixes corrective Coherent update recovery (#873) and budgets constrained by stale template replicas (#876). Upgrade the Grove controller and matching CRDs together, including installations managed separately from the Dynamo platform chart. Set spec.components[*].providerOverride.value.rollingUpdate.maxUnavailable to bound unavailable instances during a Grove rollout. Omit target; admission resolves it from the component shape, and Dynamo applies the fragment to the generated standalone clique or owning scaling group. You do not specify generated PCS names, list indices, or member clique names.

The budget counts pods for standalone cliques and complete scaling-group replicas for PCSGs. One multinode scaling-group replica includes all ranks of a distributed worker. Set the budget on the component, never on its member roles or the graph root. A budget-only override can coexist with typed topology constraints; an override that also writes topologyConstraint retains the existing restriction against mixing native and typed topology.

maxUnavailable must be a positive integer. It may exceed the component’s desired replicas or the immutable PCS template seed, including for LPX. The budget remains valid when a component scales below it or to zero; Grove limits disruption to the replicas that remain. Explicit LPX DGD replicas remain positive; omit them to let an external scaler manage live capacity, including zero. With omitted LPX replicas, admission cannot infer live serving capacity and does not emit the complete-outage warning. Under Coherent, it must also be at least the component’s effective minAvailable. A native minimum or budget does not enable Coherent; only the explicit strategy annotation does. Grove rejects rolling update configuration under OnDelete. Admission warns on creation and relevant rollout edits when an explicit budget could make every replica of a component unavailable under either RollingRecreate or Coherent. Coherent also warns when its default budget introduces this risk, including for singleton frontends. RollingRecreate retains its established warning behavior when no budget is configured. Replica-only edits warn when they introduce this risk; unrelated metadata updates and replica-only edits that leave existing risk unchanged do not repeat the warning. Pod-template edits still warn before a rollout that could take the component offline. The warning does not reject the configuration or guarantee a serving SLO. Only maxUnavailable is enabled in this subtree; other rollout knobs are rejected.

Start with minAvailable: 1 and maxUnavailable: 1 for new components unless they require a larger viable unit. Increase the budget only after measuring whether the remaining replicas meet your serving SLOs. A budget of 2 permits two unavailable instances of that component; it does not mean two instances are always removed in every step. Grove replaces unhealthy old replicas first so a corrective update can recover them; unavailable replacements limit further disruption. Grove can use additional tail steps. See Coherent capacity and disruption.

When omitted, Grove defaults the budget to minAvailable for Coherent and to 1 for RollingRecreate. Existing workloads acquire no explicit override on upgrade. Adding, changing, or removing only the budget preserves worker hashes and pod templates and does not initiate a workload rollout. Removing it restores Grove’s default for the selected strategy; Grove may persist a defaulted rollingUpdate field on the PCS. For LPX, budget edits advance the input revision and trigger child reconciliation to synchronize the policy. The PCS identity and existing pipeline requests remain unchanged.

The following single-node vLLM example uses a one-replica budget for both four prefill and eight decode replicas. Each minimum viable unit contains one prefill and one decode replica; additional tail steps account for the unequal totals. This is not a guarantee of an exact 1:2 P/D ratio at every instant. Replace the images with your compatible runtime image and create hf-token-secret if your model requires authentication. The override structure remains the same for multinode components; Dynamo routes it to the PCSG rather than a standalone clique.

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: vllm-disagg
  annotations:
    nvidia.com/grove-update-strategy: Coherent
spec:
  backendFramework: vllm
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
          name: main
    replicas: 2
    type: frontend
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model
          - Qwen/Qwen3-0.6B
          - --disaggregation-mode
          - decode
          - --kv-transfer-config
          - '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
          command:
          - python3
          - -m
          - dynamo.vllm
          envFrom:
          - secretRef:
              name: hf-token-secret
              optional: true
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          workingDir: /workspace/examples/backends/vllm
    replicas: 8
    type: decode
    providerOverride:
      apiVersion: grove.io/v1alpha1
      value:
        spec:
          minAvailable: 1
        rollingUpdate:
          maxUnavailable: 1
  - name: prefill
    podTemplate:
      spec:
        containers:
        - args:
          - --model
          - Qwen/Qwen3-0.6B
          - --disaggregation-mode
          - prefill
          - --kv-transfer-config
          - '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
          command:
          - python3
          - -m
          - dynamo.vllm
          envFrom:
          - secretRef:
              name: hf-token-secret
              optional: true
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.0
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          workingDir: /workspace/examples/backends/vllm
    replicas: 4
    type: prefill
    providerOverride:
      apiVersion: grove.io/v1alpha1
      value:
        spec:
          minAvailable: 1
        rollingUpdate:
          maxUnavailable: 1

ComponentReplicaStatus

Replica information for a single component, keyed by component name in status.components.

ComponentCheckpointStatus

Checkpoint information for a single component, keyed by component name in status.checkpoints.

Inspect a DGD

# List graph deployments (short name: dgd)
kubectl get dgd -n $NAMESPACE
# Detailed status, conditions, and events
kubectl describe dgd my-graph -n $NAMESPACE
# Per-component readiness
kubectl get dgd my-graph -n $NAMESPACE -o jsonpath='{.status.components}'

Related pages