Observability

Verify Prometheus discovery

Confirm that the application and operator monitors exist:

$
kubectl get podmonitor,servicemonitor -n dynamo-system

Then confirm the Dynamo targets are UP in Prometheus. If Prometheus is not externally available, port-forward it:

$
kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring

Open http://localhost:9090/targets and find the Dynamo targets.

Load the Dynamo dashboards

Apply the supplied dashboard ConfigMaps and the Loki data source:

$
# Application dashboard
$
kubectl apply -n monitoring \
>
  -f deploy/observability/grafana-dynamo-dashboard-configmap.yaml
$
$
# Operator dashboard
$
kubectl apply -f deploy/observability/grafana-operator-dashboard-configmap.yaml
$
$
# Logging dashboard and Loki data source
$
kubectl apply -n monitoring \
>
  -f deploy/observability/logging/grafana/loki-datasource.yaml
$
kubectl apply -n monitoring \
>
  -f deploy/observability/logging/grafana/logging-dashboard.yaml

Each dashboard ConfigMap carries the Grafana discovery label, so the Grafana sidecar imports it automatically.

Open the dashboards

Retrieve the Grafana credentials and port-forward the service:

$
export GRAFANA_USER=$(kubectl get secret -n monitoring prometheus-grafana \
>
  -o jsonpath="{.data.admin-user}" | base64 --decode)
$
export GRAFANA_PASSWORD=$(kubectl get secret -n monitoring prometheus-grafana \
>
  -o jsonpath="{.data.admin-password}" | base64 --decode)
$
echo "Grafana user: $GRAFANA_USER / password: $GRAFANA_PASSWORD"
$
$
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

Open http://localhost:3000. Under Dashboards, open the Dynamo application, operator, or logs dashboard. The application dashboard includes frontend latency and request metrics, KV router and worker metrics, GPU utilization, and pod resource usage.

Query Dynamo metrics

Use the dashboard or Prometheus to inspect a Dynamo-specific signal:

# Total frontend requests
dynamo_frontend_requests_total
# Requests waiting for a worker to start generating
sum(dynamo_frontend_stage_requests{stage=~"preprocess|route|dispatch"})
# Requests currently being processed by backend workers
sum(dynamo_component_inflight_requests{
  dynamo_component="backend",
  dynamo_endpoint="generate"
})

See the Metrics Catalog, Metric Labels, and Operator Metrics for the complete lookup information.

Enable telemetry on the worker

Add NIXL_TELEMETRY_ENABLE to each worker that should expose transfer metrics:

1
spec:
2
  services:
3
    VllmDecodeWorker:
4
      componentType: worker
5
      extraPodSpec:
6
        mainContainer:
7
          env:
8
            - name: NIXL_TELEMETRY_ENABLE
9
              value: "y"

NIXL exposes these metrics on its telemetry port, which defaults to 19090. See Environment Variables for the complete NIXL telemetry configuration.

Verify NIXL metrics

Apply the updated DGD and send traffic through the transfer path. Then confirm the NIXL metric families appear in Prometheus. The series are created only after transfers occur.

Enable JSONL logging

Set the logging variables at graph level to apply them to every component:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: vllm-agg-logging
5
spec:
6
  envs:
7
    - name: DYN_LOGGING_JSONL
8
      value: "true"
9
    - name: DYN_LOG
10
      value: "info"

See Logging for the output fields and Environment Variables for the complete configuration.

Override a component log level

Add a component-level value when one component needs more detail than the rest of the graph:

1
spec:
2
  services:
3
    Frontend:
4
      componentType: frontend
5
      extraPodSpec:
6
        mainContainer:
7
          env:
8
            - name: DYN_LOG
9
              value: "info,dynamo_runtime::system_status_server:trace"

The component value overrides the graph-level value for that container.

Inspect the logs

After applying the DGD, stream the deployment logs directly from Kubernetes:

$
kubectl logs -n dynamo-system \
>
  -l nvidia.com/dynamo-graph-deployment-name=vllm-agg-logging \
>
  --all-containers=true --prefix=true --timestamps -f

If the Loki stack and supplied dashboard are installed, open Dashboards > Dynamo Logs in Grafana to filter by DGD, namespace, and component type.

Identify the collector endpoint

Choose an OTLP endpoint reachable from the Dynamo pods. The following example uses a collector named otel-collector in the observability namespace:

http://otel-collector.observability.svc.cluster.local:4317

The collector is responsible for routing traces and logs to the configured storage backends.

Enable OTLP export

Add the OpenTelemetry settings to the DGD:

1
spec:
2
  envs:
3
    - name: OTEL_EXPORT_ENABLED
4
      value: "true"
5
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
6
      value: "http://otel-collector.observability.svc.cluster.local:4317"
7
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
8
      value: "grpc"

OTEL_EXPORT_ENABLED is the master switch for distributed traces and exported logs. For HTTP/protobuf, set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf. Use signal-specific endpoints only when traces and logs use different collectors.

Generate and inspect a trace

Apply the updated DGD and send an inference request through the frontend. In your tracing backend, find the frontend root request span and its child spans from participating Dynamo components. Structured log events emitted inside the request can include the trace and span identifiers used for correlation.

See Environment Variables for all export settings. For the internal signal flow, see Observability Architecture.

Prepare persistent storage

Create or select a persistent volume claim that can retain trace files across pod replacement. The example below expects an existing PVC named fpm-traces.

Enable FPM tracing

Mount the PVC on each worker that should write a trace and set the output prefix inside the mount:

1
spec:
2
  pvcs:
3
    - create: false
4
      name: fpm-traces
5
  services:
6
    VllmWorker:
7
      volumeMounts:
8
        - name: fpm-traces
9
          mountPoint: /var/log/dynamo
10
      extraPodSpec:
11
        mainContainer:
12
          env:
13
            - name: DYN_FPM_TRACE
14
              value: "1"
15
            - name: DYN_FPM_OUTPUT_PATH
16
              value: /var/log/dynamo/fpm

Replace VllmWorker with every worker service that should write scheduler telemetry.

Collect the trace files

Apply the updated DGD and run the workload you want to investigate. Dynamo writes a separate producer-specific file sequence for each worker. Retrieve the resulting .jsonl.gz files from the PVC for analysis.

Graceful shutdown flushes accepted records, but node loss, forced termination, queue pressure, and transport loss can create gaps. Treat these files as diagnostic data, not as a durable event log or replay input.

See Forward Pass Metrics Traces for supported topologies, capture modes, rotation, retention, and the record schema.

Check pod readiness

List the pods for a DGD and inspect their readiness state:

$
kubectl get pods -n dynamo-system \
>
  -l nvidia.com/dynamo-graph-deployment-name=<dgd-name>

A ready pod has passed the readiness probe configured by the operator.

Review the operator defaults

The frontend uses an exec readiness probe that calls /health. Workers use the system port, which defaults to 9090:

The worker startup probe allows approximately two hours for model download and engine warm-up. User-specified probes take precedence over these defaults. See the API Reference for defaults that vary by component type.

Inspect a failing probe

Describe the pod and inspect recent events and container logs:

$
kubectl describe pod -n dynamo-system <pod-name>
$
kubectl logs -n dynamo-system <pod-name> --all-containers=true

If startup is delayed by downloading model weights, prefer Model Caching over extending the probe indefinitely.

Override the probes

Set standard Kubernetes probes on the main container in the component’s podTemplate. This example extends the startup window and allows several consecutive liveness failures:

1
apiVersion: nvidia.com/v1beta1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
spec:
6
  components:
7
  - name: VllmWorker
8
    type: worker
9
    podTemplate:
10
      spec:
11
        containers:
12
        - name: main
13
          startupProbe:
14
            httpGet:
15
              path: /live
16
              port: 9090
17
            periodSeconds: 10
18
            failureThreshold: 1080
19
          readinessProbe:
20
            httpGet:
21
              path: /health
22
              port: 9090
23
            periodSeconds: 10
24
            failureThreshold: 60
25
          livenessProbe:
26
            httpGet:
27
              path: /live
28
              port: 9090
29
            periodSeconds: 10
30
            failureThreshold: 3

For a startup budget, calculate failureThreshold as the expected startup time in seconds divided by periodSeconds. Change paths or ports only when a custom engine exposes health differently.

Multinode deployments adjust probes by backend and node role. Review Multinode Deployments before overriding probes on a multinode worker.

See Health Checks for endpoint paths, responses, status codes, and active worker checks.