dynamo.frontend
OpenAI-compatible HTTP frontend, argument parsing, and pre/post-processing.
dynamo.frontend publishes 18 classes and 56 functions. Source: components/src/dynamo/frontend/__init__.py
DeepSeekPromptRenderer (class)
No summary available.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L290
Public methods
init
No summary available.
render
No summary available.
EngineFactory (class)
No summary available.
components/src/dynamo/frontend/vllm_processor.py#L1618
Public methods
init
No summary available.
chat_engine_factory
Called by Rust when a model is discovered.
FrontendArgGroup (class)
Frontend configuration parameters.
Parameters
Another process’s env (see ArgEnv). Nothing then reads the
local filesystem: --model-path is not checked, and the
AISimulate perf-model arguments, whose values name files, are not
registered.
components/src/dynamo/frontend/frontend_args.py#L423
Public methods
init
No summary available.
add_arguments
No summary available.
FrontendConfig (class)
Configuration for the Dynamo frontend.
components/src/dynamo/frontend/frontend_args.py#L194
Public methods
validate_default_chat_template_kwargs
No summary available.
validate
No summary available.
JinjaPromptRenderer (class)
No summary available.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L46
Public methods
init
No summary available.
render
No summary available.
KimiK3PromptRenderer (class)
No summary available.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L234
Public methods
init
No summary available.
render
No summary available.
PreprocessError (class)
Raised by preprocess workers for user-facing errors (e.g., n!=1).
Carries a plain message because the worker→main-process boundary pickles the exception; the main process re-raises a Dynamo-typed exception so PyO3 can route it through the proper backend-error path.
components/src/dynamo/frontend/utils.py#L219
Public methods
init
No summary available.
PreprocessResult (class)
No summary available.
components/src/dynamo/frontend/prepost.py#L84
Public methods
init
No summary available.
PromptLogprobsAccumulator (class)
Carry requested prompt logprobs through buffered chat postprocessing.
components/src/dynamo/frontend/utils.py#L126
Public methods
init
No summary available.
update
No summary available.
attach
No summary available.
PromptRenderer (class)
No summary available.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L34
Public methods
render
No summary available.
RecipeRequestError (class)
A client request rejected by the recipe encoder.
SglangEngineFactory (class)
No summary available.
components/src/dynamo/frontend/sglang_processor.py#L1280
Public methods
init
No summary available.
chat_engine_factory
Called by Rust when a model is discovered.
SglangPreprocessResult (class)
Result of SGLang preprocessing.
components/src/dynamo/frontend/sglang_prepost.py#L61
Public methods
init
No summary available.
SglangPreprocessWorkerResult (class)
Picklable return value from the SGLang preprocess worker.
components/src/dynamo/frontend/sglang_processor.py#L428
Public methods
init
No summary available.
SglangProcessor (class)
No summary available.
components/src/dynamo/frontend/sglang_processor.py#L668
Public methods
init
No summary available.
generator
Main entry point: preprocess, route, post-process a chat request.
SglangStreamingPostProcessor (class)
Streaming post-processor using SGLang parsers and HF tokenizer detokenization.
Handles:
- Incremental detokenization across tokenizer-safe boundaries
- Reasoning content extraction via SGLang ReasoningParser
- Tool call parsing via SGLang FunctionCallParser or JsonArrayParser
components/src/dynamo/frontend/sglang_prepost.py#L1032
Public methods
init
No summary available.
post_stop_token_count
Count the latest generated IDs that follow a local stop string.
The token completing the stop is kept. The count spans IDs withheld by incremental decoding and is computed on demand by re-decoding them.
process_output
Process a single engine response chunk into an OpenAI SSE choice dict.
Parameters
Dict with token_ids and optional finish_reason.
Returns
dict[str, Any] | None— OpenAI choice dict orNoneif nothing to emit yet.
StreamingPostProcessor (class)
No summary available.
components/src/dynamo/frontend/prepost.py#L1005
Public methods
init
No summary available.
process_output
No summary available.
VllmProcessor (class)
No summary available.
components/src/dynamo/frontend/vllm_processor.py#L732
Public methods
init
No summary available.
generator
Run a single request through the engine. Does pre and post processing on this machine, delegates model inference to a backend using the router.
apply_default_thinking_mode_to_template_kwargs (function)
Merge deployment thinking default unless the request already controls it.
as_error_envelope (function)
Tag an error dict so the binding reads it as an error frame.
Without _dynamo_annotated the binding reads the dict as a completion
chunk and fails with missing field `id```. The client then gets a 500 and never sees the message. The HTTP layer reads the text back off comment. See depythonize_annotatedinlib/bindings/python/rust/engine.rs“.
async_main (function)
Main async entry point for the Dynamo frontend.
Initializes the distributed runtime, configures routing, and starts the HTTP server or interactive mode based on command-line arguments.
backend_invalid_argument_to_http_error (function)
Recover a worker’s own HTTP status from a serialized backend error.
A worker that rejects a request — an unsupported sampling parameter, an
unparseable grammar — fails it with {"message": ..., "code": 4xx}.
Crossing the Rust boundary turns that into a plain Python exception whose
text is BackendInvalidArgument: {json}, so a generic except Exception reports a 500 and discards both the status and the reason.
Rust’s HTTP service already performs this recovery
(extract_backend_error_if_present in
lib/llm/src/http/service/openai.rs), but the Python chat processors do
not go through it, so they need their own. Mirror its status rules: honour
an explicit code, and otherwise fall back to 400 — the prefix itself is
the discriminator proving the argument was invalid.
Returns None when exc is not that shape, so callers fall through to
their existing handling instead of inventing a status.
build_frontend_parser (function)
The argument parser of a frontend that runs with env.
For a process that parses another frontend’s command line (the planner
reads one from the DynamoGraphDeployment spec): env defaults come from
env only, never os.environ, and nothing logs, warns or reads the
local filesystem (see FrontendArgGroup). Registers the arguments on
parser when given, else on a new ArgumentParser(add_help=False).
Returns the parser and the env var -> dest map of every argument with an
env var. Raises what the frontend raises at startup on invalid env
(ValueError, TypeError or argparse.ArgumentTypeError).
build_prompt_renderer (function)
Build the prompt renderer for a served model.
With isolate_tool_call_markers, message content that contains a
delimiter of tool_call_parser_name is tokenized without the delimiter
tokens (see encode_chat_prompt_isolating_markers); the native
DeepSeek and Kimi K3 encoders cannot do that yet and reject requests
while it is on.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L406
build_response_format_guided_decoding (function)
Build Dynamo guided decoding from OpenAI chat response_format.
build_tool_call_guided_decoding (function)
Build tool-call guidance through vLLM’s configured tool parser.
starts_in_reasoning is true when generation begins inside a reasoning
block the prompt opened and the engine does not hold the grammar back until
that block ends. The structural tag must then accept the reasoning text and
its end marker before the tool-call section.
build_tool_call_guided_decoding (function)
Build native-SGLang-like tool call constraints for guided decoding.
convert_tools (function)
Convert OpenAI tool dicts to SGLang Tool objects.
structural_tag_schema is set only for the guidance copy. Prompt and
parser copies retain SGLang’s existing omitted-strict representation.
create_parsers (function)
Create tool call and reasoning parsers for a request.
Shared by both the single-process preprocessing path and the pool path (which must recreate non-picklable parsers in the main process).
If sglang_tools is provided, reuses them; otherwise converts from
the request’s tools field.
Required and named choices use the model-specific FunctionCallParser
when the effective guidance is a structural tag. Their JSON fallback keeps
JsonArrayParser. Automatic choices use FunctionCallParser.
detect_force_reasoning_from_template (function)
Return True if the chat template auto-opens a reasoning block.
Intended to be called once at processor startup with
tokenizer.chat_template and cached on the processor.
effective_tool_strict (function)
Resolve request strictness for grammar construction.
In auto schema mode, omitted strict is schema-enforced and only an
explicit strict: false opts out. strict schema mode overrides that
request-level opt-out.
encode_chat_prompt_isolating_markers (function)
Render and tokenize the chat prompt, keeping delimiters in content as text.
Content spans that contain a tool-call delimiter are tokenized with the
delimiter-free encoder; everything the template itself emits (the tool
instructions, prior assistant tool_calls) keeps the delimiter token
ids. Prompts whose content has no delimiter tokenize exactly as
apply_chat_template(tokenize=True) does.
Fails closed: when the parser’s delimiters are not single tokens for this tokenizer, when the spans cannot be aligned with the rendered prompt, or when the tokenizer cannot be copied, the request is rejected rather than encoded with the delimiter ids.
components/src/dynamo/frontend/sglang_prompt_renderer.py#L742
extract_mm_urls (function)
Extract media and vLLM processor-cache UUIDs from chat messages.
URL-backed parts become Url variants. Image parts with no URL and an
opaque uuid become UuidOnly variants for vLLM’s multimodal
processor cache. UUID lists preserve slot order for every media type
The UUID map is None when no user UUID is present. A media content part
with neither a URL nor UUID is rejected instead of being silently dropped.
graceful_shutdown (function)
Handle graceful shutdown of the distributed runtime.
Parameters
The DistributedRuntime instance to shut down.
handle_engine_error (function)
Classify an invalid engine response and return an OpenAI-style error dict.
Called when engine_response is None or missing ‘token_ids’.
legacy_guided_decoding (function)
Convert one legacy guided-decoding constraint and its modifier to a dict.
load_frontend_route_extensions (function)
Load trusted frontend route extensions.
Each value is either a name registered under the dynamo.frontend.routes
entry-point group (preferred) or a direct module:function path.
main (function)
make_backend_error (function)
Build an OpenAI-style error dict, guarding against None/missing message.
make_internal_error (function)
Build an OpenAI-style internal error dict with request-specific fallback.
map_finish_reason (function)
named_closed_zero_arg_tool (function)
Return the named tool when its only valid argument value is {}.
nvext_extra_field_requested (function)
Return whether a request opted into a response nvext field.
parse_args (function)
Parse command-line arguments for the Dynamo frontend.
Returns
FrontendConfig— Tuple of (FrontendConfig, vllm_flags, sglang_flags, chat_processor_provider).Optional[Namespace]— The last is the resolved provider of a custom--dyn-chat-processor.
preprocess_chat_request (function)
preprocess_chat_request (function)
Preprocess a chat request using SGLang tokenizer and parser APIs.
template_force_reasoning is the static per-server flag derived from
the chat template (see detect_force_reasoning_from_template);
the effective per-request value combines it with the configured parser and
request-level thinking controls.
Synchronous — suitable for both main-process and worker-process execution.
random_call_id (function)
random_uuid (function)
read_jinja_chat_template (function)
Read a Jinja chat template using backend-specific file semantics.
render_recipe_prompt (function)
Encode a prompt on SGLang builds that provide the recipe adapter.
resolve_chat_template (function)
Return a chat template stored beside the model, or None.
Covers models (e.g. Qwen3-Omni) whose template lives in chat_template.json or chat_template.jinja rather than tokenizer_config.json, which the HF tokenizer does not merge. The backend selects native .jinja file semantics.
resolve_deepseek_spec (function)
Resolve the encoder from the downloaded checkpoint, independent of aliases.
resolve_prompt_renderer (function)
resolve_request_force_reasoning (function)
Resolve the effective force_reasoning flag for a single request.
Mirrors sglang.srt.entrypoints.openai.serving_chat._get_reasoning_from_request combined with template_manager.force_reasoning:
- opt-out families (
glm45/qwen3/kimi_k2/kimi_k3/…): on by default,chat_template_kwargs.enable_thinking=False(orthinking=Falsefor Kimi) disables it. - MiniMax-M3 defaults to adaptive, but SGLang still enables the
reasoning parser unless
chat_template_kwargs.thinking_modeis explicitly"disabled". - Mistral is enabled only when
reasoning_effortis present and not"none". - opt-in families (
deepseek-v3/gemma4): off by default, enabled bychat_template_kwargs.{thinking,enable_thinking}=True. - anything else: follow the statically-detected template default.
resolve_skip_special_tokens (function)
Honor explicit decoding options without hiding parser delimiters.
response_usage (function)
Project locally counted usage according to the client’s streaming options.
responses_history_recipients (function)
responses_history_template (function)
responses_tools_request (function)
Select renderer-only Responses metadata after resolving the template.
runtime_default_thinking_mode (function)
Read deployment-level default thinking mode from model runtime metadata.
runtime_structural_tag_excludes_reasoning (function)
Whether tool-call structural tags must leave reasoning to the backend.
Mirrors resolve_reasoning_boundary in the Rust preprocessor: the
canonical structural_tag.reasoning_boundary setting (auto by
default) decides, and auto follows the vLLM worker’s published
tool_call_structural_tag_excludes_reasoning flag. Missing or invalid
worker metadata keeps the compatibility behavior (the tag models reasoning).
runtime_structural_tag_options (function)
Read structural-tag policy from a model-card runtime config.
setup_custom_chat_engine_factory (function)
Build the chat engine factory of a custom --dyn-chat-processor.
provider is the callable parse_args resolved. It is called once,
inside the event loop, as provider(config, options), where options
is the --dyn-chat-processor-config object. It returns the async
chat_engine_factory(instance_id, mdc, routed_engine) the frontend calls
for each worker set.
setup_engine_factory (function)
When using vllm pre and post processor, create the EngineFactory that creates the engines that run requests.
setup_sglang_engine_factory (function)
When using sglang pre and post processor, create the SglangEngineFactory that creates the engines that run requests.
should_attempt_structural_tag (function)
Return whether frontend preprocessing should request a structural tag.
tool_call_marker_strings (function)
(opening, closing) delimiter of the named SGLang detector.
The detector is built the way FunctionCallParser builds the active one,
with tokenizer when its constructor accepts it: Hunyuan resolves suffixed
delimiters (<tool_calls:opensource>) from the vocabulary, and a bare
construction would name delimiters the model never emits. Surrounding
whitespace is dropped (qwen25 declares <tool_call>\n); detectors
with an empty delimiter on either side return None.
components/src/dynamo/frontend/sglang_tool_call_markers.py#L84
tool_call_marker_token_ids (function)
Map delimiter token id -> delimiter string for the active parser.
Both delimiters must encode to exactly one token; otherwise the result is
{}, the frontend keeps the text-only parser behaviour, and
encode_chat_prompt_isolating_markers rejects requests instead of
encoding them unprotected.
components/src/dynamo/frontend/sglang_tool_call_markers.py#L136
update_usage_metadata (function)
Retain known prompt/cache details across sparse backend reports.
validate_legacy_guided_decoding_constraints (function)
Reject malformed or multiple legacy guided-decoding constraints.
validate_model_name (function)
validate_model_path (function)
Validate that model-path is a valid directory on disk.
warn_if_frontend_cpu_affinity_spans_numa_nodes (function)
Log a warning if the current CPU affinity covers multiple NUMA nodes.
worker_warmup (function)
Dummy task to ensure a ProcessPoolExecutor worker is fully initialized.