NVTX Profiling
Dynamo emits NVIDIA Tools Extension (NVTX) ranges and markers, so an Nsight Systems (nsys)
capture shows where time goes inside Dynamo and not only inside the inference engine. The Rust
runtime and the Python components are annotated separately and share one runtime switch,
DYN_NVTX. Both are off by default.
Enable the Markers
Set DYN_NVTX=1 on every process you want annotated, and trace NVTX in nsys:
When the switch takes effect, the Rust runtime logs NVTX annotations enabled (DYN_NVTX) at INFO
level. The line appears only if the build includes the nvtx feature and DYN_LOG lets INFO
messages through. If it is missing, check the log level first. If other INFO messages appear and
this one does not, the build does not include the feature; see
Build Requirements.
To profile on Kubernetes, see Profiling with Nsight Operator.
Environment Variable
Since v1.7.0, DYN_NVTX also switches the Rust runtime’s markers. It replaces
DYN_ENABLE_RUST_NVTX, which Dynamo no longer reads. Earlier releases read DYN_NVTX only in the
Python components, and accepted only 0 or 1; any other value fails at import.
Build Requirements
The published wheels and images have compiled the Rust markers in since v1.6.0, so they need no
rebuild. With DYN_NVTX unset, the markers add almost no cost; see
Profiling Overhead.
If DYN_NVTX is on but the nvtx Python package is missing, the Python layer logs
DYN_NVTX is set but the nvtx package is not installed; NVTX markers are disabled and the
process starts without Python markers. The Rust markers are unaffected.
Build From Source
To build the Python bindings with the markers, include nvtx in the feature list:
To build a Rust binary for profiling, use the profiling profile, which is a release build with
debug symbols:
Scope the Cargo feature to dynamo-runtime/nvtx. A bare --features nvtx at the workspace root
also turns on the kvbm-engine markers, which ignore DYN_NVTX and fire whenever they are
compiled in.
Rust Ranges
Names are hierarchical, so a prefix filter in Nsight Systems isolates one stage. Ranges are
start/end ranges with explicit ids, so a range opened on one Tokio worker thread and closed on
another after an .await still renders as one bar.
Frontend
frontend.http.chat.first_chunk is an instantaneous marker on the first chunk that carries
generated tokens. Use it to read Time To First Token (TTFT) from the timeline. It skips the
annotation events and the role-only opening delta, which carry no generated token.
The frontend.http.* ranges and the first_chunk marker are newer than v1.7.0.
Worker
worker.ingress.prologue_sent is an instantaneous marker for the point the worker acknowledges the
response stream to the frontend. It is emitted only when that send succeeds, so a request that
progressed without it means the acknowledgement failed.
pybridge.encode_response is newer than v1.7.0. On earlier builds, the time it covers falls in
the gap before worker.egress.chunk and reads as engine time.
Python Ranges
The Python components use mm:* and mm_* names on the multimodal paths. They are emitted into
the NVTX domain dynamo, while the Rust ranges use the default domain. In the Nsight Systems GUI
they appear as separate NVTX rows. To capture only one set, pass --nvtx-domain-include=dynamo
(Python ranges) or --nvtx-domain-include=default (Rust ranges) to nsys profile. These options
filter events during collection, so nsys stats cannot apply them to an existing report.
For the SGLang worker ranges, see SGLang Multimodal.
The vLLM ranges are not listed yet. They include mm_frontend:* in the frontend’s vLLM processor
(for example mm_frontend:preprocess_chat, mm_frontend:process_inputs, and
mm_frontend:routed_engine_generate), mm_backend:build_prompt in the vLLM worker, and the shared
multimodal loaders under mm:img:*, mm:audio:*, mm:nixl:*, and mm:local:*.
Python and Rust ranges from one process land on the same timeline, together with the engine’s own NVTX ranges, such as TensorRT-LLM’s.
Reading the Timeline
The bars are annotated Dynamo work. Everything else in a request’s span falls in the gaps between bars, and no gap is a clean reading of one component.
A gap is a mix of several costs. Attributing a whole gap to the engine, or to the worker, overstates it.
- Between consecutive
worker.egress.chunkranges: the engine producing the next token, plus polling the response stream, which includes thepybridge.anext_callbar and thepybridge.encode_responseorpybridge.decode_responsebar. - Between consecutive
frontend.http.*.sse_chunkranges: worker time, the frontend receiving the next response (including thedetokenizebar), and, because an SSE range ends before the chunk is yielded, time spent by the HTTP client and the network. - Inside
frontend.http.chat.generate: dispatch of the prefill request in disaggregated serving, which is not annotated.
Range names are static, so concurrent requests produce overlapping ranges with the same name. Capture at low concurrency to follow one request from end to end.
Profiling Overhead
Annotation is cheap when DYN_NVTX is unset and costs measurable CPU time when it is on:
The figures were measured at 4096 concurrent streams on mock workers, so treat them as an
estimate on other hardware. The frontend emits about two ranges per streamed token (detokenize and sse_chunk), so at high
token rates the profiler shows up in frontend CPU time. Keep profiled runs out of measured
benchmarks.
At high token rates, nsys also writes raw trace data to /tmp at several times the size of the
final report. If that disk is slow or nearly full, write-back can stall the profiled processes for
seconds. To bound a capture, pass --duration or --capture-range to nsys profile, and keep
/tmp on fast storage with free space, for example a container started with --tmpfs /tmp.
Related
- Environment Variables lists every other observability variable.
- Request Trace Reference records per-request data for analysis that outlives a capture.
- Forward Pass Metrics Traces times the engine’s forward passes.
- Profiling with Nsight Operator covers capturing on Kubernetes.