Reinforcement Learning Integration

Experimental. This guide is for RL framework authors and rollout orchestrator maintainers who need Dynamo to serve rollouts during a training loop, return token IDs and log probabilities, expose routing or engine metadata, discover live workers, and push weight updates without restarting the serving stack. Use the frontend for rollout inference, discover vLLM workers through the read-only RL discovery API, and send lifecycle or weight updates directly to the selected worker system URL.

Backend Support

Experimental means the RL integration surfaces are still converging around real framework usage. Treat nvext.token_data, the documented nvext.extra_fields values, and /v1/rl/workers for RL-enabled vLLM workers as the API-shaped surfaces to build against first. Treat engine_data, SGLang uploaded meta_info, direct /engine/ route bodies, custom route names, and backend-native response shapes as backend-specific escape hatches that can change independently.

Choose an Interface

The discovery API runs on a dedicated frontend listener. It is not mounted on the main frontend port, and it does not proxy /engine/ calls. The request-plane URL in its response is used internally to query route descriptors; an RL orchestrator should use the returned system_url for administration.

vLLM Happy Path

This is the shortest path for wiring an RL orchestrator to a vLLM rollout worker: start the frontend discovery listener, start an RL-enabled worker, discover its system URL, run a token-in rollout, then pause, update weights, and resume generation.

$
DYN_ENABLE_RL=true DYN_RL_PORT=8001 python -m dynamo.frontend

$
DYN_SYSTEM_PORT=8081 python -m dynamo.vllm \
>
  --model Qwen/Qwen3-0.6B \
>
  --enable-rl

$
curl http://localhost:8001/v1/rl/workers

$
curl http://localhost:8000/v1/completions \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "prompt": "",
>
    "max_tokens": 32,
>
    "temperature": 0,
>
    "logprobs": 5,
>
    "prompt_logprobs": 5,
>
    "nvext": {
>
      "token_data": [151644, 8948, 198],
>
      "extra_fields": ["completion_token_ids", "prompt_logprobs"]
>
    }
>
  }'

$
worker_url=http://10.0.0.12:8081
$
$
curl --fail-with-body "$worker_url/engine/pause_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"mode": "keep", "clear_cache": false}'
$
$
curl --fail-with-body "$worker_url/engine/update_weights_from_disk" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"model_path": "/models/checkpoint-42", "weight_version": "42"}'
$
$
curl --fail-with-body "$worker_url/engine/resume_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{}'

Frontend Feature Support

The OpenAI-compatible completion routes provide the current token-in/token-out interface.

See NVIDIA Request Extensions for the complete nvext reference.

Backend RL flags also select engine-specific behavior:

• On vLLM, --enable-rl registers the discovery and administration routes, selects processed log probabilities, and prevents model generation_config.json sampling overrides from changing RL token-in requests marked with nvext.token_data. Explicit request sampling values still apply.
• On SGLang, --enable-rl enables out-of-band meta_info upload and the /engine/call_tokenizer_manager passthrough route. SGLang also exposes control routes such as /engine/control/update_weights_from_disk, but SGLang workers do not currently register with /v1/rl/workers.

Token-In/Token-Out Example

Send pre-tokenized input and request token IDs plus prompt and completion log probabilities:

$
curl http://localhost:8000/v1/completions \
>
  -H 'Content-Type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "prompt": "",
>
    "max_tokens": 32,
>
    "temperature": 0,
>
    "logprobs": 5,
>
    "prompt_logprobs": 5,
>
    "nvext": {
>
      "token_data": [151644, 8948, 198],
>
      "extra_fields": ["completion_token_ids", "prompt_logprobs"]
>
    }
>
  }'

The relevant response fields have this shape:

1
{
2
  "choices": [
3
    {
4
      "index": 0,
5
      "text": "...",
6
      "logprobs": {}
7
    }
8
  ],
9
  "nvext": {
10
    "completion_token_ids": [9707, 11],
11
    "prompt_logprobs": [
12
      null,
13
      {"8948": {"logprob": -0.42}},
14
      {"198": {"logprob": -0.31}}
15
    ]
16
  }
17
}

To use the chat route while retaining pre-tokenized input, provide the normal messages field and set nvext.token_data to the complete token sequence that the engine should receive. The frontend uses token_data instead of rendering and tokenizing messages.

Routed Expert Data

Opt into routed-expert data per request:

1
{
2
  "model": "my-moe-model",
3
  "prompt": [1, 2, 3],
4
  "max_tokens": 16,
5
  "nvext": {
6
    "extra_fields": ["completion_token_ids", "routed_experts"]
7
  }
8
}

The payload format is backend-specific:

• vLLM-compatible builds return an object containing base64 data, shape, dtype, and start. The start offset identifies the first returned routing row in the full prompt-plus-completion sequence.
• SGLang builds whose async_generate API supports return_routed_experts return the engine’s base64 string. Start SGLang with --enable-return-routed-experts to request capture. Builds that omit this engine argument do not return the field.
• TensorRT-LLM does not currently return routed-expert data through this frontend extension.

Use nvext.engine_data only when the orchestrator must consume other backend-specific data. The named completion_token_ids, prompt_logprobs, and routed_experts fields provide more stable contracts.

Upload SGLang Metadata

SGLang can upload the final cumulative meta_info for each choice to any filesystem supported by the installed fsspec backend. This path keeps large log probability tensors, routed-expert data, and custom metadata out of the HTTP response.

Start the worker with RL support. Add the routed-expert flag only when the SGLang engine build supports it:

$
DYN_SYSTEM_PORT=8081 python -m dynamo.sglang \
>
  --model-path Qwen/Qwen3-0.6B \
>
  --enable-rl \
>
  --enable-return-routed-experts

Set a unique upload directory for each request:

1
{
2
  "model": "Qwen/Qwen3-0.6B",
3
  "prompt": [151644, 8948, 198],
4
  "max_tokens": 32,
5
  "logprobs": 5,
6
  "nvext": {
7
    "metadata_upload": {
8
      "url": "s3://rl-rollouts/run-42/request-7"
9
    }
10
  }
11
}

The worker writes choice_0.msgpack.zst, choice_1.msgpack.zst, and so on beneath the URL. Each file contains a Zstandard-compressed MessagePack object:

1
{
2
  "schema_version": 1,
3
  "metadata": {
4
    "id": "...",
5
    "finish_reason": {"type": "stop"},
6
    "output_token_logprobs": [],
7
    "output_top_logprobs": [],
8
    "routed_experts": "..."
9
  }
10
}

Install the matching fsspec extra for remote storage, such as fsspec[s3] for S3. The worker also requires msgspec and zstandard. Local URLs such as file:///tmp/rollouts/request-7 are useful for development.

When metadata_upload is present, SGLang uploads only the final cumulative meta_info for each choice and omits inline log probabilities and routed-expert metadata. The final response waits for the upload. An upload failure fails the request, so use a durable destination and a unique URL to avoid overwriting another request’s choice_<index>.msgpack.zst objects.

Discover RL Workers

RL discovery currently covers RL-enabled vLLM workers. Start the frontend discovery listener and a vLLM worker with its system server enabled:

$
DYN_ENABLE_RL=true DYN_RL_PORT=8001 python -m dynamo.frontend

$
DYN_SYSTEM_PORT=8081 python -m dynamo.vllm \
>
  --model Qwen/Qwen3-0.6B \
>
  --enable-rl

Query the dedicated discovery port:

$
curl http://localhost:8001/v1/rl/workers

The response describes each live worker and probes it for the routes available in that process:

1
{
2
  "namespace": "dynamo",
3
  "workers": [
4
    {
5
      "namespace": "dynamo",
6
      "component": "backend",
7
      "endpoint": "rl",
8
      "instance_id": 12345,
9
      "transport": {"tcp": "10.0.0.12:1234/..."},
10
      "request_plane_url": "dyn://dynamo.backend.rl",
11
      "system_url": "http://10.0.0.12:8081",
12
      "model": "Qwen/Qwen3-0.6B",
13
      "routes": [
14
        "get_weight_version",
15
        "liveness_probe",
16
        "pause_generation",
17
        "resume_generation",
18
        "update_weights_from_disk"
19
      ]
20
    }
21
  ]
22
}

Treat routes as the capability list for that worker instead of assuming every engine exposes the same methods. Optional features such as Low-Rank Adaptation (LoRA) add routes only when enabled. Discovery can also return a worker with an error and an empty route list when its descriptor probe times out or fails.

The discovery listener uses these environment variables:

Call vLLM Engine Routes Directly

Read system_url and routes from the discovery response, then call the selected vLLM worker. Send a JSON object even when the route does not require arguments:

$
curl http://10.0.0.12:8081/engine/liveness_probe \
>
  -H 'Content-Type: application/json' \
>
  -d '{}'

For a vLLM weight update cycle, pause the selected worker, call one of its advertised weight-update routes, validate the result, and resume generation. Install jq before running this example. The exit trap attempts to resume the worker if the update or another command fails:

$
set -euo pipefail
$
$
worker_url=http://10.0.0.12:8081
$
$
resume_generation() {
>
  response=$(curl --fail-with-body "$worker_url/engine/resume_generation" \
>
    -H 'Content-Type: application/json' \
>
    -d '{}')
>
  jq -e '.status == "ok"' <<<"$response" >/dev/null
>
}
$
$
pause_response=$(curl --fail-with-body "$worker_url/engine/pause_generation" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"mode": "keep", "clear_cache": false}')
$
jq -e '.status == "ok"' <<<"$pause_response" >/dev/null
$
trap resume_generation EXIT
$
$
update_response=$(curl --fail-with-body "$worker_url/engine/update_weights_from_disk" \
>
  -H 'Content-Type: application/json' \
>
  -d '{"model_path": "/models/checkpoint-42", "weight_version": "42"}')
$
jq -e '.status == "ok"' <<<"$update_response" >/dev/null
$
$
resume_generation
$
trap - EXIT

Route request bodies and response fields are engine-specific. A callback can return {"status":"error"} with HTTP 200, while an exception in the callback produces HTTP 500. Check both the HTTP status and the JSON result before advancing the rollout state.

SGLang direct administration uses SGLang-specific routes and response shapes. For example, call /engine/control/update_weights_from_disk directly when you already know the worker URL, or call /engine/call_tokenizer_manager with {"method":"update_weights_from_disk", ...} for tokenizer-manager passthrough behavior.

The /v1/rl/workers endpoint is read-only. It intentionally does not expose /v1/rl/engine or /v1/rl/engines proxy routes, which prevents an accidental frontend fan-out of a mutating engine operation.

Register a Custom Engine Route

Register an asynchronous Python callback on the worker’s DistributedRuntime. The route name is appended to /engine/ on the system server:

1
from typing import Any
2
3
4
async def set_rollout_state(body: dict[str, Any]) -> dict[str, Any]:
5
    rollout_id = body.get("rollout_id")
6
    if not isinstance(rollout_id, str) or not rollout_id:
7
        return {"status": "error", "message": "rollout_id is required"}
8
9
    await engine.set_rollout_state(rollout_id)
10
    return {"status": "ok", "rollout_id": rollout_id}
11
12
13
runtime.register_engine_route("rl/set_rollout_state", set_rollout_state)

Enable the system server with DYN_SYSTEM_PORT, then call the route directly:

$
curl http://worker-host:8081/engine/rl/set_rollout_state \
>
  -H 'Content-Type: application/json' \
>
  -d '{"rollout_id": "run-42-step-7"}'

register_engine_route makes the HTTP route callable but does not automatically advertise it in the RL discovery response. When extending an RL-enabled vLLM worker, register and advertise the route together with the shared helper:

1
from dynamo.common.rl import register_rl_routes
2
3
4
register_rl_routes(
5
    runtime,
6
    handler.rl_route_registry,
7
    {"rl/set_rollout_state": set_rollout_state},
8
    enable_dispatch=handler.config.enable_rl,
9
)

The next GET /v1/rl/workers probe includes rl/set_rollout_state in that worker’s routes list.