> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # ThunderAgent Program Scheduler **Experimental.** NVIDIA Dynamo includes a native ThunderAgent plugin for program-aware admission and worker placement. It runs inside the frontend and is enabled through router-policy YAML. It ports the scheduler from the [ThunderAgent](https://arxiv.org/abs/2602.13692) paper (Kang et al., 2026). ## Native Frontend Plugin To implement your own admission policy, see [Write an Admission Control Plugin](/dynamo/dev/knowledge-base/modular-components/router/write-an-admission-control-plugin). Use this path for aggregated serving through `dynamo.frontend`. It requires a Dynamo build that includes the `thunderagent` builtin plugin; older wheels and images do not recognize that type. The plugin is included in the builtin catalog without a separate ThunderAgent build feature. Admission behavior remains unchanged until the plugin is selected in YAML. Custom catalogs that already register `thunderagent` must remove that registration; duplicate type names fail startup. Save the following as `thunderagent.yaml`: ```yaml request_classifier: type: thunderagent worker_selection: aggregated: thunderagent instances: - name: thunderagent type: thunderagent ``` Start the frontend against your existing aggregated workers, using the same discovery configuration as the workers: ```bash python3 -m dynamo.frontend \ --router-mode kv \ --router-policy-config thunderagent.yaml \ --router-session-affinity-ttl-secs 1800 \ --router-session-affinity-mode soft ``` Send a stable `X-Dynamo-Session-ID` on every turn of an agent's reasoning and tool-use chain. Dynamo also accepts the [native agent headers](/dynamo/dev/agents/session-i-ds#native-agent-headers). Requests without session context pass through admission unchanged. At the end of a session, a dedicated minimal request with `X-Dynamo-Session-Final: true` releases the retained program state; otherwise idle retention expires it. Select both plugin roles. The classifier owns the program table and delays admission when a session is busy or paused. When it releases a request, it supplies a preferred worker/rank. The worker selector honors that target if it is eligible, falling back to the least-loaded eligible candidate otherwise. Soft session affinity allows this repacking; hard affinity would constrain the request before the selector can move it. The `Sent` event records the worker actually selected. ```mermaid flowchart LR A[Request and session ID] --> B[ThunderAgent classifier] B -->|Released request and preferred worker| C[Dynamo policy queue] C --> D[ThunderAgent worker selector] D --> E[Aggregated worker] E -->|Progress and lifecycle events| B ``` The classifier uses each worker/rank's published GPU KV-block capacity and a logical token estimate from active and retained programs. It observes context growth during generation, then pauses and repacks programs between turns. This estimate is not physical KV occupancy and does not deduplicate shared prefixes or include host/offload capacity. Running generation is allowed to finish. ### Native Plugin Parameters Set admission parameters under `request_classifier.parameters`. Leave worker-selection parameters empty. Unknown parameter names and invalid values fail startup. | Parameter | Default | Meaning | |---|---|---| | `pause_threshold` | `0.95` | Per-worker usage fraction that starts pausing programs. | | `pause_target` | `0.80` | Usage fraction the pause cycle drains toward. | | `resume_hysteresis` | `0.10` | Headroom below the pause threshold required for normal resume. | | `resume_timeout_seconds` | `1800` | Maximum deferral before forced release on a live worker. This does not reject the request. | | `session_retention_seconds` | `1800` | Retention period for an idle, continuing program. | | `scheduler_interval_seconds` | `5` | Interval between pressure and resume decisions. | | `acting_token_weight` | `1` | Capacity weight while the agent is doing tool work. | | `acting_decay_tau_seconds` | `1` | Acting-token decay half-life used for timeout fallback placement. | | `buffer_per_program` | `100` | Token headroom reserved for each active program. | | `max_tracked_requests` | `10000` | Independent limits on tracked requests and retained programs. Requests fail classification at the request limit. New programs fail at the program limit if no idle program can be evicted. | The default deferral timeout permits waits of roughly 30 minutes under pressure. Set client deadlines to match the workload's acceptable wait. Router policy classes and queues still apply after classification releases a request. Program state is local to each frontend's router instance. This is not a cluster-wide admission budget; keep a session's turns on the same frontend when using multiple replicas. The native plugin has been exercised with aggregated vLLM workers. Standalone EPP and disaggregated deployment guidance are outside this example. The native plugin's classifier and worker selector register together in Dynamo's builtin catalog. It does not expose the Python prototype's soft-demotion controls, priority boosts, host-cache budget, or separate status endpoints described below. ### Native Plugin Logs The plugin logs its interval, deferral timeout, and tracking limit at INFO when admission is enabled. A WARN event reports the number of programs forcibly resumed after reaching the deferral timeout. These requests continue processing; they are not rejected. Timeout warnings are aggregated into one event per scheduler tick. To see scheduler state changes, enable DEBUG for the `thunderagent` log target before starting the frontend: ```bash export DYN_LOG='info,thunderagent=debug' ``` Each changed tick reports counts of active, paused, and marked-for-pause programs, waiting requests, and tracked requests. Unchanged ticks are silent. Log snapshots use stored counts without scanning programs or requests, and log emission happens after releasing the scheduler lock. ## Python Router Prototype The remaining sections describe the older standalone Python implementation. Run that prototype from a source checkout. Its CLI flags and observability endpoints differ from the native plugin above. `dynamo.thunderagent_router` is a standalone Dynamo router that schedules at the granularity of an agent run — the whole `LLM turn → tool call → next turn` loop — instead of individual requests. It wraps Dynamo's native KV router and adds a program-level scheduler with tool-boundary pause/resume on top of KV-aware routing, porting the scheduler from the [ThunderAgent](https://arxiv.org/abs/2602.13692) paper (Kang et al., 2026). ## The Problem Agentic workloads (SWE-bench, browser-use, anything with a tool loop) make many short LLM calls separated by non-GPU work: `docker exec`, `pytest`, `curl`, waiting on a subagent. Between turns the agent's KV cache stays resident, holding blocks while doing nothing. A request-level router (vLLM's, SGLang's, Dynamo's stock `KvRouter`) sees each turn but not the agent behind it, which costs you two ways: - **Cache-occupancy blowup.** With N agents at step K, the working set is `N × step_K_context`, most of it idle between turns. The engine evicts useful blocks under pressure or refuses admission, and every next turn pays a re-prefill tax. - **No tool-boundary backpressure.** The router can't defer a hot session at a natural pause point — it can only cancel in-flight requests or queue them, both worse than waiting until the agent is between turns. ## The Scheduler The algorithm groups requests by `program_id` (the header-derived `session_id`) and runs an outer scheduler that moves each program through `(REASONING | ACTING) × (ACTIVE | PAUSED)`. A program enters ACTING at a tool boundary. Under memory pressure the scheduler pauses ACTING programs — logically, with no decode preemption — so the engine is free to evict their KV. When utilization drops it resumes the smallest-token programs first, BFD-packing them back under threshold. The payoff is working-set accounting that counts programs rather than requests, plus pause/resume aimed at tool boundaries rather than arbitrary tokens. This is an in-path Dynamo service that owns a `KvRouter` directly and registers as a model handler, so there is no extra proxy hop, and it reads real `prompt_tokens + completion_tokens` off each response rather than estimating token counts from raw bytes. ### Scheduler Tick A single background task runs every `--scheduler-interval-seconds` (default `5.0`). Each tick takes a capacity snapshot and runs three phases in a fixed order: ```text _apply_soft_demotes → _greedy_resume → _pause_until_safe ``` Resume runs **before** pause on purpose (upstream ThunderAgent ordering): a program paused this tick cannot resume until the next tick, which prevents a program from being paused and immediately resumed within one tick. ### Tool-Boundary Pause/Resume Semantics - **Pause** is logical. The scheduler picks the smallest ACTING programs on an over-threshold worker first and pauses them; if no ACTING candidate exists it marks the smallest REASONING program for pause at its next tool boundary. There is no decode preemption — a paused program's in-flight turn is allowed to finish, and the program is held out of admission until a later tick resumes it. - **Resume** is greedy and BFD-packed. When a worker has headroom (see the control loop below), the scheduler resumes the smallest-token paused programs first, fitting each back under threshold and accounting for `buffer_per_program`. Resumed requests get a transient priority boost so they re-enter ahead of fresh admissions, and a forced-resume cap (`--resume-timeout-seconds`) guarantees no program is starved indefinitely. ### Program Lifetime A program is created on its first turn, keyed by `session_id`. Public session identity is carried in headers such as `x-dynamo-session-id`, `x-dynamo-parent-session-id`, and `x-dynamo-session-final`. Program bookkeeping must still be bounded by router policy, such as idle expiry or token-weight decay. ## Utilization-Driven Control Loop Pause/resume is driven by per-worker utilization — the program working set as a fraction of the worker's retention budget. With SGLang HiCache enabled, Dynamo reads the worker's published GPU KV and host HiCache capacities and uses their sum. The host tier is included so native GPU-to-host spill can happen before this scheduler pauses programs. Mooncake is excluded: it is conditional content-addressed storage, not guaranteed per-program retention. The loop has three bands: - At or above `pause-threshold`, the worker is over-subscribed; the tick pauses ACTING programs until utilization falls back to `pause-target`. - In the `[soft-demote-threshold, pause-threshold)` band, programs are soft-demoted (a negative priority jump) but not paused — early backpressure before a hard pause is needed. - Resume only fires once utilization has dropped at least `resume-hysteresis` below `pause-threshold`, so the loop does not oscillate between pause and resume on the threshold boundary. | Flag | Env var | Default | Description | |---|---|---|---| | `--pause-threshold` | `DYN_THUNDERAGENT_PAUSE_THRESHOLD` | `0.95` | Working-set fraction of the retention budget that fires a pause cycle. | | `--soft-demote-threshold` | `DYN_THUNDERAGENT_SOFT_DEMOTE_THRESHOLD` | `0.80` | Soft-demote band start (negative priority jump in `[soft, pause)`). | | `--pause-target` | `DYN_THUNDERAGENT_PAUSE_TARGET` | `0.80` | Setpoint that pause cycles drive utilization back down to. Must be `<= pause-threshold`. | | `--resume-hysteresis` | `DYN_THUNDERAGENT_RESUME_HYSTERESIS` | `0.10` | Headroom below `pause-threshold` required before any resume. | | `--resume-priority-boost` | `DYN_THUNDERAGENT_RESUME_PRIORITY_BOOST` | `1.0` | Priority seconds added to a request that just resumed. | | `--resume-timeout-seconds` | `DYN_THUNDERAGENT_RESUME_TIMEOUT_SECONDS` | `1800.0` | Forced-resume cap. Mirrors ThunderAgent's `_wait_for_resume`. | | `--scheduler-interval-seconds` | `DYN_THUNDERAGENT_SCHEDULER_INTERVAL_SECONDS` | `5.0` | Scheduler tick period. | | `--soft-demote-priority-jump` | `DYN_THUNDERAGENT_SOFT_DEMOTE_PRIORITY_JUMP` | `-2.0` | Priority seconds applied to soft-demoted programs. | | `--acting-token-weight` | `DYN_THUNDERAGENT_ACTING_TOKEN_WEIGHT` | `1.0` | Multiplier on `token_total` for ACTING programs in the **pause-side** working set. | | `--acting-decay-tau-seconds` | `DYN_THUNDERAGENT_ACTING_DECAY_TAU_SECONDS` | `1.0` | Tau for exponential decay of ACTING tokens in the **resume-side** working set. | > **Constraint:** `pause-target <= pause-threshold`. The service rejects configs that violate it (along with `0 <= resume-hysteresis <= pause-threshold` and `0 <= soft-demote-threshold <= pause-threshold`). All `KvRouter` flags from `dynamo.router` (`--router-temperature`, `--use-kv-events`, `--router-track-output-blocks`, …) are also accepted and forwarded. ## Architecture ```text ┌─────────────────────────────────────────────────────────────┐ │ dynamo.frontend (HTTP + auth + tracing sink) │ └────────────────────┬────────────────────────────────────────┘ │ chat completions, with session headers ▼ ┌─────────────────────────────────────────────────────────────┐ │ dynamo.thunderagent_router (this service) │ │ - ProgramTable: session_id → ProgramState │ │ - admission gate: before_request → was_paused? │ │ - scheduler loop (every scheduler_interval_seconds): │ │ _apply_soft_demotes → _greedy_resume → _pause_until_safe│ │ - sticky worker pin from program.assigned_worker_id │ │ - after_request: real-token accounting │ └────────────────────┬────────────────────────────────────────┘ │ KvRouter.generate ▼ ┌─────────────────────────────────────────────────────────────┐ │ KvRouter (in-process; subscribes to KV events + FPM) │ └────────────────────┬────────────────────────────────────────┘ │ per-worker dispatch ▼ ┌─────────────────────────────────────────────────────────────┐ │ dynamo.vllm (N workers; FPM publisher, KV events publisher)│ └─────────────────────────────────────────────────────────────┘ ``` ## Observability The router emits INFO logs for lifecycle transitions and DEBUG logs for per-request route decisions. These lines let you confirm that a request went through ThunderAgent, whether it was a one-off passthrough or a program-scoped turn, and which worker hint or selected worker was used. **Request routing** — logged at DEBUG once per request: ```text thunderagent.route path= ... ``` For program-scoped traffic, the line includes `program`, `prompt_tokens`, `worker_hint`, `waited_seconds`, `was_paused`, `soft_demoted`, and `priority_jump`. When the first response chunk carries backend worker attribution, the router also logs at DEBUG: ```text thunderagent.route_selected program= worker= source=first_chunk ``` **Lifecycle transitions** — logged when a program is created, paused, resumed, or terminated: ```text thunderagent.program created program= ... thunderagent.program paused program= reason= ... thunderagent.program resumed program= worker= ... thunderagent.program terminated program= ... ``` **Request completion** — logged at DEBUG after token accounting runs: ```text thunderagent.request_complete program= prompt_tokens= completion_tokens= ... ``` The scheduler also emits a per-tick INFO summary on each side of the control loop, so both pause and resume activity are visible at INFO without enabling DEBUG. **Pause side** — logged when a worker pauses or marks any program in a tick: ```text scheduler.tick worker= paused= marked= util= -> ``` `paused` is the number of ACTING programs paused this tick, `marked` is the number of REASONING programs marked for pause at their next tool boundary, and `util=X -> Y` is the worker utilization before and after the pause cycle. **Resume side** — logged when a worker resumes any program in a tick: ```text scheduler.tick resumed= still_paused= ``` `resumed` is the number of programs resumed this tick and `still_paused` is the size of the paused table afterward. This line is symmetric to the pause-side summary; before it existed, pause was observable at INFO but resume was only visible at DEBUG, leaving a gap when reconstructing a control-loop cycle from INFO logs alone. ### Status and Metrics Endpoints ThunderAgent serves two Dynamo runtime endpoints alongside `generate`: - `.thunderagent_router.status` returns scheduler state, active/paused program counts, per-program lifecycle state, per-replica utilization keyed `:`, and request counters (including `unpinned_turns`). - `.thunderagent_router.metrics` returns counters and gauges shaped for quick operational checks, including created/ended programs, admitted/paused requests, pause/resume totals, forced resumes, `unpinned_turns_total`, and per-replica utilization. These are Dynamo runtime endpoints, not public OpenAI HTTP routes. Use them from another Dynamo component or a small runtime client connected to the same namespace. ### Response Route Proof For response-level debugging, request `nvext.extra_fields=["engine_data"]`. Only when that field is requested, ThunderAgent adds a `thunderagent` object under `nvext.engine_data` on generated chunks, with fields such as `handled_by`, `path`, `program_id`, `was_paused`, `waited_seconds`, `priority_jump`, `assigned_worker_hint`, `assigned_dp_rank_hint`, and `selected_worker_id` with `selected_dp_rank` when worker attribution is available. For per-request tracing (token counts, cache hits, worker placement), the router also integrates with [Agent Tracing](/dynamo/dev/agents/agent-tracing#enable-output): set `DYN_REQUEST_TRACE=1` on the frontend to land a `request_end` record per LLM call. Harness tool-event spans are separate: they require `DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT` plus a configured publisher. ## Reproducing with upstream Harbor and Pi The maintained end-to-end path uses upstream Harbor to create SWE-bench containers and Pi with the Dynamo provider inside each container. The complete source build, ThunderAgent arm, stock KV arm, stable-session setup, and scaling procedure live in the [`thunderagent_router` README](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/thunderagent_router/README.md#harborpi-ab-walkthrough). ## References - ThunderAgent paper: [arxiv.org/abs/2602.13692](https://arxiv.org/abs/2602.13692) - Upstream ThunderAgent reference: [HaoKang-Timmy/ThunderAgent](https://github.com/HaoKang-Timmy/ThunderAgent) - Pi Dynamo provider: [ai-dynamo/agent-plugins](https://github.com/ai-dynamo/agent-plugins/tree/main/pi-plugin) - Dynamo KV router: [Router Guide](/dynamo/dev/knowledge-base/modular-components/router/router-guide) - [Session IDs](/dynamo/dev/agents/session-i-ds), [Agent Tracing](/dynamo/dev/agents/agent-tracing), and [Agent Hints](/dynamo/dev/agents/agent-hints)