> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v1.6.1) (default): https://docs.dynamo.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.dynamo.nvidia.com/dynamo/dev/llms.txt
> - v1.6.1: https://docs.dynamo.nvidia.com/dynamo/v1.6.1/llms.txt
> - v1.6.0: https://docs.dynamo.nvidia.com/dynamo/v1.6.0/llms.txt
> - v1.5.1: https://docs.dynamo.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.dynamo.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.dynamo.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.dynamo.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.dynamo.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.dynamo.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.dynamo.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.dynamo.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.dynamo.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.dynamo.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.dynamo.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.dynamo.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.dynamo.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.dynamo.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.dynamo.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.dynamo.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.dynamo.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.dynamo.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.dynamo.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Prioritize Requests

By the end of this guide, requests that you mark as higher priority are served ahead of lower-priority requests when workers are busy. Use it when interactive agent turns (the one a user is waiting on) compete with background subagents for the same workers.

Priority only changes anything under contention. If a request can be routed and scheduled immediately, there is no queue to reorder. Each layer below has its own switch, so turn on the layers you need.

![A high-priority interactive request is released ahead of earlier background requests from the router queue. Priority can act in three places, each with its own switch: the router queue, the backend engine scheduler, and KV cache eviction.](/dynamo/_fern-files/dynamo.docs.buildwithfern.com/7034a37a2c26e0f6921381ae6b5f33db9974864c07da6e7259f5f5164b3c1c64/assets/img/priority-scheduling.svg)

## Send a Priority

Add `nvext.agent_hints` to the request body:

**`Request with priority hints`**

```json title="Request with priority hints"
{
    "model": "my-model",
    "messages": [
        { "role": "user", "content": "Summarize this incident." }
    ],
    "nvext": {
        "agent_hints": {
            "priority": 10,
            "strict_priority": 1
        }
    }
}
```

| Hint              | Effect                                                                                                                                                            |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `priority`        | Soft priority. Higher values mean higher priority. The router uses it to order its queue, and supported backends use it for engine scheduling and cache eviction. |
| `strict_priority` | Router-only pending-queue tier, an unsigned integer. A higher tier always goes ahead of a lower tier. It is not forwarded to the backend engine.                  |

For HTTP requests you can send the same values as `x-dynamo-request-priority` and `x-dynamo-request-strict-priority`. Header values override the matching `nvext.agent_hints` fields. Both methods need the frontend to accept NvExt inputs: they have no effect if you start it with `DYN_DISABLE_FRONTEND_NVEXT=true`.

Sending a priority changes nothing until you turn on a layer below. The `agg_agent.sh` launcher enables priority-aware cache eviction only; it does not enable the router queue or engine priority scheduling.

## Turn On Priority in Each Layer

Priority can act in three places. They are configured separately and are additive.

### 1. Router queue

The router holds a request in a queue while every eligible worker is busy, then releases requests in priority order. Queueing is off until you enable it in a router policy file. Save this as `policy.yaml`:

```yaml
policy_classes:
  - name: standard
    quantum: 1
    queue_policy: fcfs
    prefill_busy_threshold_frac: 1.0
```

Start the frontend with KV routing and the policy file:

```bash
python -m dynamo.frontend \
  --router-mode kv \
  --router-policy-config policy.yaml
```

A request waits while every eligible worker's active prefill tokens exceed `prefill_busy_threshold_frac` times its `max_num_batched_tokens`. Lower the fraction to queue sooner. Keep `queue_policy` set to `fcfs` or `wspt`: the default, `edf`, ignores `priority` and `strict_priority` when it orders the queue. See [Policy-Class Queues](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#policy-class-queues) for the full schema, including `wspt` ordering and per-worker queue limits.

> **Note**
>
> The older `--router-queue-threshold` and `--router-queue-policy` flags are deprecated and are planned for removal in v1.7. Use the policy file instead.

### 2. Backend engine

Each engine has its own switch. Dynamo converts the priority to the polarity the engine expects.

#### vLLM

Start vLLM with `--scheduling-policy priority`. Do not negate `priority` in client code; Dynamo forwards it with the polarity vLLM expects.

#### SGLang

Start SGLang with `--enable-priority-scheduling`. Higher Dynamo values become higher SGLang scheduling priority. Dynamo rejects the inverted SGLang flag. See [Agents on SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang#priority-scheduling).

#### TensorRT-LLM

Per-request engine scheduling priority is not exposed through Dynamo. Priority still affects router queueing before dispatch.

### 3. KV cache eviction

When GPU cache fills, an engine can evict low-priority entries first. On SGLang, start the worker with `--radix-eviction-policy priority`. This only matters under memory pressure. See [Priority-based KV cache eviction](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang#priority-based-kv-cache-eviction).

Support by backend:

| Feature                       |   vLLM  | SGLang | TensorRT-LLM |
| ----------------------------- | :-----: | :----: | :----------: |
| Priority-aware routing        |   Yes   |   Yes  |      Yes     |
| Engine priority scheduling    |   Yes   |   Yes  |      No      |
| Priority-based cache eviction | Planned |   Yes  |    Planned   |

## Check That It Works

Use a benchmark that sends different `priority` values on individual requests. For AIPerf, use a version with per-request `extra` payload support. Older versions only support global `--extra-inputs`, which cannot produce mixed-priority tiers in one run.

For router priority:

* Send a fixed number of requests per priority tier, with the same model, input length, output length, and streaming mode.
* Run at enough load for requests to wait. Watch [`dynamo_frontend_router_queue_pending_requests`](/dynamo/dev/reference/observability/metrics-catalog#router-queue-metrics) and confirm it is above zero during the measured window.
* Configure the backend flag separately if you want to measure engine scheduling and not only the router queue.

Expected result: higher priority values get better TTFT under contention. If lower values win, check whether the client, benchmark harness, or gateway negated the priority before it reached Dynamo.

## Troubleshooting

| Symptom                                                                   | Check                                                                                                                                                                                                                                                  |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Priority has no visible effect.                                           | Confirm requests actually enter the router queue. Confirm the backend priority flag is on if you expect engine-level scheduling.                                                                                                                       |
| Lower numeric values appear to win.                                       | Do not negate `priority` for vLLM. Dynamo normalizes backend polarity.                                                                                                                                                                                 |
| The router queue never becomes non-empty.                                 | Lower `prefill_busy_threshold_frac`, increase offered load, or check the SGLang `max_num_batched_tokens` caveat in [Router Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#tuning-guidelines). |
| Priority works through the frontend but not through a Kubernetes gateway. | Confirm the gateway path preserves `nvext`, and use Dynamo v1.2.0 or later.                                                                                                                                                                            |
| AIPerf cannot assign a priority per request.                              | Use an AIPerf build with per-request `extra` payload support.                                                                                                                                                                                          |

## How Priority Works

### Router queue ordering

A policy class defaults to the `edf` queue policy, which dispatches the earliest due time first, then keeps arrival order, and ignores `priority` and `strict_priority`. To let these hints reorder the router queue, set the class's `queue_policy` to `fcfs` or `wspt`. Without a policy file, the synthetic class uses `--router-queue-policy`, `fcfs` by default. These policies order queued requests by this key:

```text
(strict_priority, due_at, configured_policy_key)
```

The strict tier is compared first. Within a tier, every request with a request classifier's `due_at` goes ahead of every request without one, and then the configured policy applies. Sustained deadline-bearing traffic can therefore starve non-deadline requests in the same class.

The `fcfs` policy treats `priority` as a positive arrival-time bump: higher values move the request earlier. Negative values are clamped to zero for router queueing, so a request cannot be pushed behind normal first-come ordering. The `wspt` policy orders by `(1 + priority_jump) / scheduling_cost_tokens` and optimizes average TTFT.

### What priority does not do

Priority is not Kubernetes `PriorityClass`, GPU preemption, or hard admission control, and it does not reserve capacity. Strict priority applies only to requests already waiting in one scheduler queue. It does not preempt admitted work, order requests across router replicas, or guarantee engine execution order. A new arrival that is eligible can still be admitted directly while others wait.

## Other Hints

Two more hints change serving behavior without being about priority:

| Hint                  | Description                                                                                                                                                            |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `osl`                 | Expected output length in tokens. The router uses it for output block tracking and load-balancing accuracy when `--router-track-output-blocks` is enabled.             |
| `speculative_prefill` | When `true`, Dynamo prefills the predicted prefix of the next turn after the current turn completes, to warm the KV cache. On SGLang this is a `max_tokens=1` prefill. |

Hints and session IDs are separate. A session ID is passive identity for traces and optional policies. Hints are active serving intent. Neither enables session affinity; see [Maintain session affinity](/dynamo/dev/agents/maintain-session-affinity).

## Version Notes

| Capability                                                    | Availability            |
| ------------------------------------------------------------- | ----------------------- |
| Router priority queue and backend priority plumbing           | Dynamo v1.0.0 and later |
| Higher `priority` means higher priority across the API        | Dynamo v1.1.0 and later |
| EPP and Inference Gateway forwarding fixes for priority hints | Dynamo v1.2.0 and later |

## Related

* [NVIDIA Request Extensions](/dynamo/dev/additional-resources/nvidia-request-extensions-nvext#agent-hints)
* [Router Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning)
* [vLLM Reference Guide](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/reference-guide#priority-scheduling)
* [Agents on SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang)