Profile with Nsight Operator

NVIDIA Nsight Operator adds Nsight Systems (nsys) to the pods you select when they are created. It starts and stops capture windows on request, and each profiled pod uploads one report per profiled process to S3 or to Nsight’s in-cluster storage. Its CLI and web UI are called Nsight Cloud.

Nsight’s Dynamo Quick Start Documentation covers installing and running the operator. This page adds only the parts that need Dynamo knowledge:

• which pod labels select each Dynamo component;
• a profile config with process patterns and nsys arguments for each component;
• the NVTX settings that add Dynamo’s own ranges to the reports;
• limitations when profiling Dynamo.

You can use the published Dynamo runtime images from NGC without a custom build, and the DynamoGraphDeployment (DGD) needs no new fields.

These instructions use Nsight Operator 26.3. If your cluster runs an older version, upgrade to 26.3 first.

Who Does What

Nsight’s multitenant quickstart defines three roles. To profile a Dynamo deployment, each role does the following:

Commands and reports take different paths:

By default, Nsight adds no privileged flag, capabilities, or host access to profiled pods, and nsys runs as the container’s user. To keep it that way, do not turn on GPU metrics sampling (the nsys option --gpu-metrics-devices). Nsight’s GPU metrics setup needs extra privileges: it adds a collector DaemonSet whose pods run with privileged: true (or the SYS_ADMIN capability). Tracing CUDA API calls and GPU kernels, as the worker configs below do, needs no extra privileges.

Try Nsight’s Dynamo Helper First

Nsight ships a Dynamo helper, nsight_operator_dynamo.py, with a Dynamo quick start. On a cluster where the operator already runs with auto-provisioning, its configure command (Option B) gives the fastest first look. Adjust it as follows:

• Pass lowercase component types, such as --components frontend or --components prefill decode. The helper compares them with the lowercase nvidia.com/dynamo-component-type label, so capitalized component names such as Frontend select no pod.
• Pass --deployment-name <your DGD>. The helper selects every pod whose name contains that string, so pick a name that no other DGD’s name contains.
• Set --nsys-profile-args "-t cuda,nvtx,osrt --python-sampling=true". The helper’s default adds --pytorch=autograd-nvtx (see Limitations).
• Leave out --gpu-metrics. It installs Nsight’s GPU metrics collector: a DaemonSet whose pods run with privileged: true, plus a profile config that runs nsys --gpu-metrics-devices=all in those pods.

For example, to profile the frontend and the aggregated workers of one DGD:

python3 nsight_operator_dynamo.py -n <namespace> configure \
    --deployment-name <dgd> \
    --components frontend worker \
    --nsys-profile-args "-t cuda,nvtx,osrt --python-sampling=true"

The helper then prints the next steps: deploy the DGD or recreate its pods, then start a capture, stop it, and list the reports with nsight_operator.py.

Select Components

Select pods with the labels the Dynamo operator already sets on every pod. Always include nvidia.com/dynamo-graph-deployment-name, so that other DGDs in the namespace are not matched.

Check the labels of your pods with:

kubectl get pods -n <namespace> -l nvidia.com/dynamo-graph-deployment-name=<dgd> --show-labels

Write a Profile Config

Apply one NsightOperatorProfileConfig for each DGD you profile, before you deploy that DGD; only pods created after the config exists are profiled. This config follows Nsight’s per-tenant storage example and uses the field names of Nsight Operator 26.3. Replace my-dgd everywhere with your DGD name, my-namespace with your namespace, and team-s3 with your storage config. Keep one rule for each component you want to record.

apiVersion: nvidia.com/v1
kind: NsightOperatorProfileConfig
metadata:
  name: nsys-my-dgd
  namespace: my-namespace
spec:
  nsightToolConfigs:
  - name: my-dgd-cpu           # frontend and planner: OS runtime calls, Python stacks, NVTX
    nsightToolArgs: "-t nvtx,osrt --python-sampling=true"
    opentelemetryTrace: true
  - name: my-dgd-gpu           # workers: adds CUDA API calls and every kernel
    nsightToolArgs: "-t cuda,nvtx,osrt --cuda-graph-trace=node --python-sampling=true"
    opentelemetryTrace: true
  injectionRules:              # rule names must be unique in the namespace
  - name: my-dgd-frontend
    nsightToolConfigRef: my-dgd-cpu
    nsightTags: [my-dgd]                # literal text; the same tag on every rule
    cloudStorageConfigRef: team-s3      # omit to use Nsight's in-cluster storage
    podObjectSelector:
      matchLabels:
        nvidia.com/dynamo-graph-deployment-name: my-dgd
        nvidia.com/dynamo-component-type: frontend
    processIncludePatterns: ['^(.*/)?python(3([.][0-9]+)?)? -m dynamo[.]frontend( .*)?$']
  - name: my-dgd-planner
    nsightToolConfigRef: my-dgd-cpu
    nsightTags: [my-dgd]
    cloudStorageConfigRef: team-s3
    podObjectSelector:
      matchLabels:
        nvidia.com/dynamo-graph-deployment-name: my-dgd
        nvidia.com/dynamo-component-type: planner
    processIncludePatterns: ['^(.*/)?python(3([.][0-9]+)?)? -m dynamo[.]planner( .*)?$']
  - name: my-dgd-decode
    nsightToolConfigRef: my-dgd-gpu
    nsightTags: [my-dgd]
    cloudStorageConfigRef: team-s3
    podObjectSelector:
      matchLabels:
        nvidia.com/dynamo-graph-deployment-name: my-dgd
        nvidia.com/dynamo-component-type: decode
    processIncludePatterns: ['^(.*/)?python(3([.][0-9]+)?)? -m dynamo[.](vllm|sglang|trtllm)( .*)?$']

• OpenTelemetry. Both tool configs enable OpenTelemetry trace mirroring.
• Tags. Every rule uses the same literal tag, the DGD name, so one window profiles every selected component over the same period. A tag holds one profiling session at a time. For separate windows per component, give each rule its own tag.
• Names. Nsight looks up tool configs by name across the whole namespace, so give each profile config in a namespace its own tool-config names. Do not name the config dynamo-profile-config, which Nsight’s helper writes and deletes.
• Patterns. They match Dynamo’s launch commands, python -m dynamo.<module> and python3 -m dynamo.<module>, so helper commands such as python3 -c ... are not wrapped.
• Fork tracing. The config leaves out --trace-fork-before-exec=true, which Nsight’s template sets. The nsys documentation warns that it can make the application crash or deadlock.

Settings per Component

• --cuda-graph-trace=node shows every kernel inside CUDA graphs and costs the most. For lower overhead, use --cuda-graph-trace=graph, which shows each graph launch as one item.
• Set the NVTX variables in the component’s container env, or in the DGD’s spec.env to set them on every component, before the pods start.
• Keep each window at 300 seconds or less; Nsight Systems does not officially support longer runs.

Capture

Get nsight_operator.py from the NGC resources of the same release as the operator. It needs Python 3.12 or newer. Install its dependencies from the unpacked bundle:

pip install -r requirements.txt

Deploy the DGD after the profile config, wait until every selected pod is ready, and start your load. Then run:

NS=my-namespace
T=my-dgd
python3 nsight_operator.py autoconfigure -n $NS
python3 nsight_operator.py --tag $T status                # one collection agent per selected pod
python3 nsight_operator.py --tag $T session-begin --title $T
python3 nsight_operator.py --tag $T profiler-start --delay 120 --duration 60
python3 nsight_operator.py --tag $T ls                    # repeat until every pod has a report
python3 nsight_operator.py --tag $T download -o ./reports
python3 nsight_operator.py --tag $T session-end

--delay gives the load time to warm up. The coordinator starts and stops the window itself, so the CLI does not need to stay connected. Download before session-end: without --session <UUID>, ls and download use the tag’s active session. To download later, pass the session ID that session-begin printed. Reports appear after the delay, the window, and the upload; a 30-second window on one GPU worker produced a report of about 100 MB. Each file name carries the tag, the process, and the pod name. An otlp-converter entry in status or ls comes from Nsight’s OpenTelemetry converter, not from a Dynamo pod. Open the reports with an Nsight Systems version at least as new as the nsys that the operator injects.

Profile One Replica of a Component

A rule selects every replica of a component: the injector decides when a pod is created, before the pod has its final name, so a rule cannot pick one replica by name. To profile one replica, give each pod its own tag and start the window on one pod’s tag. This uploads to the namespace’s in-cluster storage, which Nsight creates with auto-provisioning; it was not tested with an S3 storage config such as team-s3, so leave out cloudStorageConfigRef on this rule. Change the profile config above in two places:

  nsightToolConfigs:
  - name: my-dgd-gpu
    nsightToolArgs: "-t cuda,nvtx,osrt --cuda-graph-trace=node --python-sampling=true"
    opentelemetryTrace: true
    containerEnv:              # per-pod tags have no storage of their own; upload to the namespace's default storage
    - name: NSIGHT_CLOUD_STORAGE_CONFIG_FILE
      value: /mnt/nv/secrets/nsight-cloud-storage/default.yaml
  injectionRules:
  - name: my-dgd-decode
    nsightToolConfigRef: my-dgd-gpu
    nsightTags: ['${HOSTNAME}']  # each pod's tag is its own pod name
    podObjectSelector:
      matchLabels:
        nvidia.com/dynamo-graph-deployment-name: my-dgd
        nvidia.com/dynamo-component-type: decode
    processIncludePatterns: ['^(.*/)?python(3([.][0-9]+)?)? -m dynamo[.](vllm|sglang|trtllm)( .*)?$']

Then run the commands in Capture with one pod’s name as the tag. Set T to that pod name instead of my-dgd, and do not rerun the T=my-dgd line:

kubectl get pods -n $NS -l nvidia.com/dynamo-graph-deployment-name=my-dgd,nvidia.com/dynamo-component-type=decode
T=<pod name>
python3 nsight_operator.py --tag $T status    # only this pod

The window records only that pod, and its report goes to Nsight’s in-cluster storage. Other pods selected by the rule stay idle until you start a window on their own tags.

Profile One Pod with kubectl exec

To profile a single pod, you can also run nsys inside it with kubectl exec, without nsight_operator.py. The pod must already be injected, which means a profile config selected it when the pod was created. The injected nsys is at /mnt/nv/bin/nsight-systems/target-linux-x64/nsys on x86_64 nodes and /mnt/nv/bin/nsight-systems/target-linux-sbsa-armv8/nsys on Arm nodes. Dynamo’s container is named main:

POD=my-dgd-0-worker-abcde
NSYS=/mnt/nv/bin/nsight-systems/target-linux-x64/nsys
kubectl exec -n $NS $POD -c main -- $NSYS sessions list    # the session name, such as coordinated-default-1
kubectl exec -n $NS $POD -c main -- $NSYS start --session=coordinated-default-1
kubectl exec -n $NS $POD -c main -- $NSYS stop --session=coordinated-default-1

nsys stop waits until the report is written, which can take up to a minute for a GPU worker, and then prints its path after Generated:, for example /tmp/tmpkro1yevt/<report>.nsys-rep. That file exists only inside the pod’s main container: the operator does not upload it, nsight_operator.py ls and download do not list it, and it is lost if the container restarts. Copy it out as soon as nsys stop returns:

kubectl exec -n $NS $POD -c main -- cat /tmp/tmpkro1yevt/<report>.nsys-rep > ./reports/worker.nsys-rep

Limitations

• Apply the profile config before the pods start. Only pods created after the config exists are profiled. After you change rules or arguments, recreate the pods.
• Use literal tags, or ${HOSTNAME} for one tag per pod. Other placeholders, such as {NVDT_POD_NAME} or ${NVDT_POD_FULLNAME}, silently put the pod on the default tag.
• Leave out --pytorch=autograd-nvtx. Nsight’s helper adds it by default. It can create a CUDA context on GPU 0 for every rank, which can shrink the KV cache.
• Leave DYN_NVTX unset on workers. On Dynamo 1.6.0, DYN_NVTX=1 makes python -m dynamo.vllm fail at import under nsys, because some NVTX color names need matplotlib, which the runtime images do not ship. SGLang workers load the same module.
• Keep profiled runs out of measured benchmarks. During a 30-second window with node-level CUDA graph tracing on one vLLM worker, end-to-end latency rose by about 18%; a frontend-only window changed it by less than 1%.
• Use one profiler per process. If your image already starts Dynamo under nsys, use an image without that wrapper.
• TensorRT-LLM on Blackwell GPUs: -t cuda can use hardware CUDA trace. If a worker crashes in a device-to-device copy under tracing, use -t cuda-sw,nvtx,osrt. This was seen with nsys outside Kubernetes.
• If a pod is not profiled, follow Nsight’s sidecar injection checks. Setting logOutput: /tmp/injector.log on a rule writes the process hook’s log inside the pod.