SLA-Driven Profiling with DynamoGraphDeploymentRequest

Overview

Dynamo provides automated SLA-driven profiling through DynamoGraphDeploymentRequests (DGDR). Instead of manually running profiling scripts, you declare your performance requirements and let the Dynamo Operator handle profiling and deployment automatically.

Key Benefits:

• Declarative: Specify SLAs, not implementation details
• Automated: No manual job setup or result processing
• Integrated: Seamlessly works with Dynamo Operator
• Production-Ready: Generates optimized configurations with SLA planner

This document covers:

• Technical details of online vs offline profiling
• Profiling process internals (GPU usage, measurements, interpolation)
• Direct script usage for advanced scenarios
• Comprehensive troubleshooting

Support Matrix

Specifically, the profiler sweeps over the following parallelization mapping for prefill and decode:

Using DGDR for Profiling (Recommended)

The recommended way to profile models is through DGDRs. Sample configurations are provided in deploy/:

Available Samples:

• profile_sla_dgdr.yaml: Standard profiling with AIPerf on real engines
• profile_sla_aic_dgdr.yaml: Fast profiling with AI Configurator simulation
• profile_sla_moe_dgdr.yaml: MoE model profiling

The Dynamo Operator automatically:

1. Discovers GPU resources (cluster-scoped operators only)
2. Runs profiling (AIPerf on real engines or AI Configurator simulation)
3. Generates optimal DGD configuration with SLA planner
4. Deploys the DGD to your cluster

See the Quick Start Guide for prerequisites and detailed instructions.

Hardware Configuration

Hardware parameters have sensible defaults and are optional - you can override them if needed:

1
profilingConfig:
2
  config:
3
    # Override hardware defaults if needed
4
    hardware:
5
      minNumGpusPerEngine: 1
6
      maxNumGpusPerEngine: 8
7
      numGpusPerNode: 8
8
9
    # Only needed when using AI Configurator (sweep.useAiConfigurator: true)
10
    sweep:
11
      aicSystem: h200_sxm  # GPU type for AI Configurator (h100_sxm, h200_sxm, etc.)

Automatic GPU Discovery (Optional Feature)

Cluster-scoped operators can optionally enable automatic GPU discovery to detect hardware from cluster nodes. When enabled, hardware config is auto-detected and overrides any manually specified values.

1
spec:
2
  enableGpuDiscovery: true

This feature is only available with cluster-scoped operators (namespaceRestriction.enabled=false) as it requires cluster-wide node access permissions. It is not available for namespace-restricted operators.

Profiling Method

1. Hardware Setup: Uses defaults or user-specified hardware configuration. Optionally, cluster-scoped operators can enable automatic GPU discovery to detect specifications from cluster nodes.
2. Identify Sweep Ranges: Automatically determine minimum and maximum number of GPUs per engine. Minimum is determined by the model size and GPU VRAM. Maximum is set to one node for dense model and 4 nodes for MoE models.
3. Parallelization Mapping Sweep: Use the input ISL and OSL, test the performance of the engines with different parallelization mappings.
   • For dense models, we test different TP sizes for both prefill and decode.
   • For MoE models (SGLang), we evaluate both TEP and DEP as candidates for prefill and decode.
   • Prefill:
     • TP/TEP: We measure TTFT with batch size = 1 (assuming ISL is long enough to saturate compute) without KV reuse.
     • DEP: Attention uses data parallelism. We send a single burst with total concurrency attention_dp_size × attn_dp_num_req_ratio (defaults to 4) and compute the reported TTFT as time_to_first_token.max / attn_dp_num_req_ratio from the AIPerf summary of that burst. This stabilizes measurements when the first batch may launch before all requests arrive.
   • Decode: Since the ITL (or iteration time) is relevant with how many requests are in-flight, we measure the ITL under different number of in-flight requests. The range of the number of in-flight requests is from 1 to the maximum number of requests that the kv cache of the engine can hold. To measure the ITL without being affected by piggy-backed prefill requests, the script will enable kv-reuse and warm up the engine by issuing the same prompts before measuring the ITL. Since the kv cache is sufficient for all the requests, it can hold the kv cache of the pre-computed prompts and skip the prefill phase when measuring the ITL. However, for MoE models, this is not guaranteed because the kv cache in different attention DP ranks is different. We are working on framework-side change to fix this issue. For example, the below plot shows the decode parallelization mapping sweep results for H100 for deepseek-ai/DeepSeek-R1-Distill-Llama-8B.
4. Recommendation: Selects optimal parallelization mapping for prefill and decode that achieves the highest per GPU throughput while adhering the SLA on TTFT and ITL. Specifically, the profiler will choose the point (or a point on the curve for decode) that is left to the vertical red dashed line that represents the SLAs while has the highest y coordinate (throughput per GPU).
5. In-Depth Profiling on the Recommended P/D Engine: After finding the best TP size for prefill and decode, the script will then interpolate the TTFT with ISL and ITL with active KV cache and decode context length. This is to provide a more accurate estimation of the performance when ISL and OSL changes and will be used in the sla-planner.
   • Prefill: Measures TTFT and throughput per GPU across different input lengths with batch size=1.
   • Decode: Measures ITL and throughput per GPU under various KV cache loads and decode context lengths. The active kv usage determines the complexity of the memory-bounded attention kernel while the active kv usage divided the average context length determines the complexity of the computation bound MLP kernel. For example, the below figure shows the ITL of DS-Distilled Llama 8b model on H100 TP4. The ITL grows near-linearly with active kv usage under a fixed context length. And the slope increases as the context length decreases.

To run the parallelization mapping sweep and the in-depth profiling on the recommended P/D engine, the profiler need to know the engine’s forward pass time with different loads. There are two ways to achieve this: run AIPerf on real engines or use AI Configurator to run simulations.

AIPerf on Real Engines

Profiles your model by creating real test deployments in Kubernetes and measuring their performance.

Characteristics:

• Duration: 2-4 hours
• Accuracy: Highest (real measurements)
• GPU Requirements: Full access to test different parallelization mappings
• Backends: vLLM, SGLang, TensorRT-LLM

DGDR Configuration:

1
profilingConfig:
2
  config:
3
    sweep:
4
      useAiConfigurator: false  # Default

AI Configurator Simulation

Uses performance simulation to rapidly estimate optimal configurations without running real deployments.

Characteristics:

• Duration: 20-30 seconds
• Accuracy: Estimated (may have errors for unusual configurations)
• GPU Requirements: None
• Backends: TensorRT-LLM only (vLLM/SGLang coming soon)

DGDR Configuration:

1
profilingConfig:
2
  config:
3
    sweep:
4
      useAiConfigurator: true
5
      aicSystem: h200_sxm          # GPU system type
6
      aicHfId: Qwen/Qwen3-32B      # HuggingFace model ID
7
      aicBackendVersion: "0.20.0"

Supported Configurations:

For the current list of supported models, systems, and backend versions, see the AI Configurator documentation.

To check from the command line: aiconfigurator cli --help

Currently supports:

• Backends: TensorRT-LLM (versions 0.20.0, 1.0.0rc3, 1.0.0rc6)
• Systems: H100 SXM, H200 SXM, B200 SXM, GB200 SXM, A100 SXM
• Models: Wide range including GPT, Llama, Mixtral, DeepSeek, Qwen, and more

Output Format

After profiling, the DGDR status contains:

1. Recommended Configuration: Optimal TP for prefill and decode
2. Performance Data: Interpolation models for SLA planner
3. Generated DGD: Complete deployment manifest

Example Recommendations:

Suggested prefill TP:4 (TTFT 48.37 ms, throughput 15505.23 tokens/s/GPU)
Suggested decode TP:4 (ITL 4.83 ms, throughput 51.22 tokens/s/GPU)

Interactive Configuration Selection WebUI

When running the profiler with --pick-with-webui, an interactive web interface is launched that allows you to visually explore profiling results and manually select configurations.

Features:

• Interactive Charts: Visualize prefill TTFT, decode ITL, and GPU hours analysis with hover-to-highlight synchronization between charts and tables
• Pareto-Optimal Analysis: The GPU Hours table shows pareto-optimal configurations balancing latency and throughput
• DGD Config Preview: Click “Show Config” on any row to view the corresponding DynamoGraphDeployment YAML
• GPU Cost Estimation: Toggle GPU cost display to convert GPU hours to cost ($/1000 requests)
• SLA Visualization: Red dashed lines indicate your TTFT and ITL targets

Selection Methods:

1. GPU Hours Table (recommended): Click any row to select both prefill and decode configurations at once based on the pareto-optimal combination
2. Individual Selection: Click one row in the Prefill table AND one row in the Decode table to manually choose each

Example DGD Config Output:

When you click “Show Config”, you’ll see a DynamoGraphDeployment configuration like:

1
# DynamoGraphDeployment Configuration
2
# Prefill: 1 GPU(s), TP=1
3
# Decode: 4 GPU(s), TP=4
4
# Model: Qwen/Qwen3-32B-FP8
5
# Backend: trtllm
6
apiVersion: nvidia.com/v1alpha1
7
kind: DynamoGraphDeployment
8
spec:
9
  services:
10
    PrefillWorker:
11
      subComponentType: prefill
12
      replicas: 1
13
      extraPodSpec:
14
        mainContainer:
15
          args:
16
          - --tensor-parallel-size=1
17
    DecodeWorker:
18
      subComponentType: decode
19
      replicas: 1
20
      extraPodSpec:
21
        mainContainer:
22
          args:
23
          - --tensor-parallel-size=4

Usage:

$
python -m benchmarks.profiler.profile_sla \
>
  --backend trtllm \
>
  --config path/to/disagg.yaml \
>
  --pick-with-webui \
>
  --use-ai-configurator \
>
  --model Qwen/Qwen3-32B-FP8 \
>
  --aic-system h200_sxm \
>
  --ttft 200 --itl 15

Once you have selected a configuration, the full DynamoGraphDeployment CRD will be saved in your output folder as config_with_planner.yaml.

The WebUI launches on port 8000 by default (configurable with --webui-port).

Output Performance Plots

The profiler will generate the following plots to better visualize the performance data:

Parallelization Mapping Sweep Plots:

• prefill_performance.png: TTFT vs Parallelization Mapping size
• decode_performance.png: ITL vs Parallelization Mapping size and in-flight requests

Note these two plots are based on the input ISL and OSL.

In-Depth Profiling for the Recommended P/D Engine Plots:

• selected_prefill_interpolation/prefill_ttft_interpolation.png: TTFT vs ISL for the recommended prefill engine
• selected_prefill_interpolation/prefill_throughput_interpolation.png: Throughput vs ISL for the recommended prefill engine
• selected_decode_interpolation/decode_itl_interplation.png: ITL vs KV usage and context length for the recommended decode engine
• selected_decode_interpolation/decode_throughput_interpolation.png: Throughput vs KV usage and context length for the recommended decode engine

Output Interpolation Data

The profiler generates .npz files to store the performance data for the recommended P/D engine:

Prefill Interpolation (selected_prefill_interpolation/raw_data.npz):

• prefill_isl: 1D array of input sequence lengths tested
• prefill_ttft: 1D array of TTFTs (ms) at each ISL
• prefill_thpt_per_gpu: 1D array of throughput (tokens/s/GPU) at each ISL

Decode Interpolation (selected_decode_interpolation/raw_data.npz):

• max_kv_tokens: Total KV tokens capacity in decode engine
• x_kv_usage: 1D array of active KV usage percentages [0, 1]
• y_context_length: 1D array of average context lengths tested
• z_itl: 1D array of ITLs (ms) at each (KV usage, context length) point
• z_thpt_per_gpu: 1D array of throughput (tokens/s/GPU) at each point

DGDR Configuration Reference

This section provides detailed explanations of all DGDR profilingConfig options. The DGDR controller passes this configuration to the profiler script, which is defined in benchmarks/profiler/utils/profiler_argparse.py.

Configuration Structure

All profiler configuration goes under spec.profilingConfig.config:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: my-deployment
5
spec:
6
  model: "Qwen/Qwen3-0.6B"         # High-level: model to deploy
7
  backend: vllm                    # High-level: inference backend
8
9
  profilingConfig:
10
    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"  # Required
11
    configMapRef:                  # Optional: base DGD config
12
      name: my-config
13
      key: disagg.yaml
14
15
    config:                        # Profiler configuration
16
      sla: { ... }
17
      hardware: { ... }
18
      sweep: { ... }               # AIC settings go here (aicSystem, aicHfId, etc.)
19
      planner: { ... }
20
21
  deploymentOverrides:             # Optional
22
    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"

SLA Configuration (Required)

Define your performance requirements and workload characteristics:

1
profilingConfig:
2
  config:
3
    sla:
4
      isl: 3000      # Average input sequence length (tokens)
5
      osl: 150       # Average output sequence length (tokens)
6
      ttft: 200.0    # Target Time To First Token (milliseconds)
7
      itl: 20.0      # Target Inter-Token Latency (milliseconds)

What these control:

• ISL/OSL: Based on your expected traffic patterns
• TTFT: First token latency target (lower = more GPUs needed, affects prefill engine)
• ITL: Token generation latency target (lower = more GPUs needed, affects decode engine)
• Trade-offs: Tighter SLAs require more GPU resources

Hardware Configuration (Optional)

Control GPU search space and constraints:

1
profilingConfig:
2
  config:
3
    hardware:
4
      minNumGpusPerEngine: 2      # if not provided, will automatically determine based on model and VRAM size
5
      maxNumGpusPerEngine: 8      # Maximum GPUs to test
6
      numGpusPerNode: 8            # GPUs per node (for multi-node MoE)
7
      gpuType: h200_sxm              # GPU type hint

When to use:

• minNumGpusPerEngine: Skip small TP sizes if your model is large
• maxNumGpusPerEngine: Limit search space or work around constraints (e.g., AIC attention heads)
• numGpusPerNode: Determine the upper bound of number of GPUs per node for dense models and configure Grove for multi-node MoE engines.
• gpu_type: Informational, auto-detected by controller

Sweep Configuration (Optional)

Control profiling behavior:

1
profilingConfig:
2
  config:
3
    sweep:
4
      useAiConfigurator: false              # Use offline profiling (default: false)
5
      prefillInterpolationGranularity: 16   # Samples for prefill TTFT curve
6
      decodeInterpolationGranularity: 6     # Samples for decode ITL curve

Use cases:

• useAiConfigurator: Set to true for 20-30 second profiling (TensorRT-LLM only)
• prefillInterpolationGranularity: How many samples to benchmark for prefill TTFT curve (lower = faster but may be less accurate)
• decodeInterpolationGranularity: How many samples to benchmark for decode ITL curve (lower = faster but may be less accurate). Since ITL interpolation is a 3d plot and takes longer to run, we default to a smaller number of samples. Increasing this value might quadratically increase the profiling time.

AI Configurator Configuration (Required if useAiConfigurator: true)

Configure AI Configurator profiling mode:

1
profilingConfig:
2
  config:
3
    sweep:
4
      useAiConfigurator: true
5
      aicSystem: h200_sxm              # GPU system: h100_sxm, h200_sxm, b200_sxm, gb200_sxm, a100_sxm
6
      aicHfId: Qwen/Qwen3-32B         # Huggingface model id
7
      aicBackendVersion: "0.20.0"     # TensorRT-LLM version: 0.20.0, 1.0.0rc3

Supported configurations: See AI Configurator documentation

Planner Configuration (Optional)

Pass arguments to the SLA planner:

1
profilingConfig:
2
  config:
3
    planner:
4
      planner_min_endpoint: 2                    # Minimum endpoints to maintain
5
      planner_adjustment_interval: 60            # Adjustment interval (seconds)
6
      planner_load_predictor: linear             # Load prediction method

Model Cache PVC (Advanced)

For large models, you can use a pre-populated PVC containing model weights instead of downloading from HuggingFace. This is useful when:

• The model is not publicly available on HuggingFace
• You want to avoid repeated downloads during profiling
• You have a shared model cache across your cluster

1
profilingConfig:
2
  config:
3
    deployment:
4
      modelCache:
5
        pvcName: "model-cache"                        # Name of PVC containing model weights (required)
6
        pvcPath: "hub/models--deepseek-ai--DeepSeek-R1"  # Subpath within PVC (optional)
7
        mountPath: "/opt/model-cache"                 # Mount path in container (optional, default: /opt/model-cache)

Requirements:

• The PVC must exist in the same namespace as the DGDR
• The model weights must be accessible at {mountPath}/{pvcPath}

Engine Configuration (Auto-configured)

The controller automatically sets these from high-level fields:

1
# You specify:
2
spec:
3
  model: "Qwen/Qwen3-0.6B"
4
  backend: vllm
5
6
# Controller auto-injects into config:
7
profilingConfig:
8
  config:
9
    deployment:
10
      model: "Qwen/Qwen3-0.6B"       # From spec.model
11
    engine:
12
      backend: vllm                  # From spec.backend
13
      config: /path/to/configmap     # From spec.profilingConfig.configMapRef (if provided)

You should not manually set deployment.model or engine.backend in profilingConfig.config - they are automatically injected from the high-level fields.

Complete Example: AIPerf on Real Engines

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: vllm-dense-online
5
spec:
6
  model: "Qwen/Qwen3-0.6B"
7
  backend: vllm
8
9
  profilingConfig:
10
    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"
11
    config:
12
      sla:
13
        isl: 3000
14
        osl: 150
15
        ttft: 200.0
16
        itl: 20.0
17
18
      hardware:
19
        minNumGpusPerEngine: 1
20
        maxNumGpusPerEngine: 8
21
22
      sweep:
23
        useAiConfigurator: false
24
25
  deploymentOverrides:
26
    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"
27
28
  autoApply: true

Complete Example: AI Configurator Simulation

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: trtllm-aic-offline
5
spec:
6
  model: "Qwen/Qwen3-32B"
7
  backend: trtllm
8
9
  profilingConfig:
10
    profilerImage: "nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.6.1"
11
    config:
12
      sla:
13
        isl: 4000
14
        osl: 500
15
        ttft: 300.0
16
        itl: 10.0
17
18
      sweep:
19
        useAiConfigurator: true
20
21
      aic:
22
        system: h200_sxm
23
        model_name: QWEN3_32B
24
        backend_version: "0.20.0"
25
26
  deploymentOverrides:
27
    workersImage: "nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.6.1"
28
29
  autoApply: true

Complete Example: MoE Model

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: sglang-moe
5
spec:
6
  model: "deepseek-ai/DeepSeek-R1"
7
  backend: sglang
8
9
  profilingConfig:
10
    profilerImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
11
    config:
12
      sla:
13
        isl: 2048
14
        osl: 512
15
        ttft: 300.0
16
        itl: 25.0
17
18
      hardware:
19
        numGpusPerNode: 8
20
        maxNumGpusPerEngine: 32
21
22
      engine:
23
        isMoeModel: true       # Enable MoE profiling mode
24
25
  deploymentOverrides:
26
    workersImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
27
28
  autoApply: true

Troubleshooting

Profiling Takes Too Long

Solution 1: Use AI Configurator for rapid profiling (TensorRT-LLM only):

1
sweep:
2
  useAiConfigurator: true

Solution 2: Reduce search space:

1
config:
2
  sweep:
3
    minNumGpus: 4  # Skip TP1, TP2
4
    maxNumGpus: 8  # Don't test beyond TP8

SLA Cannot Be Met

Symptoms: Profiler reports no configuration meets targets

Solutions:

1. Relax SLA targets (increase TTFT/ITL)
2. Add more GPU resources
3. Try a different backend
4. Use a smaller model

AI Configurator: Attention Head Constraint Error

Symptoms: Profiling fails with error:

AssertionError: num_heads <N> should be divisible by tp_size <M> and the division result should be >= 4

Cause: AI Configurator requires ≥4 attention heads per GPU. Small models with few heads cannot use high TP sizes.

Affected Models:

• Qwen3-0.6B (16 heads): Max TP = 4 ❌ Fails at TP=8
• GPT-2 (12 heads): Max TP = 3
• Most models <1B parameters: May hit this constraint

Solution: Limit maxNumGpusPerEngine in your DGDR:

1
profilingConfig:
2
  profilerImage: "nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.6.1"
3
  config:
4
    hardware:
5
      maxNumGpusPerEngine: 4  # For Qwen3-0.6B (16 heads / 4 = max TP of 4)
6
    sweep:
7
      useAiConfigurator: true
8
      aicSystem: h200_sxm
9
      aicHfId: Qwen/Qwen3-0.6B

Calculate Max TP: max_tp = num_attention_heads / 4

Note: This is an AI Configurator limitation. Online profiling doesn’t have this constraint.

Image Pull Errors

Symptoms: ErrImagePull or ImagePullBackOff

Solution: Ensure image pull secrets are configured:

$
kubectl create secret docker-registry nvcr-imagepullsecret \
>
  --docker-server=nvcr.io \
>
  --docker-username='$oauthtoken' \
>
  --docker-password=<NGC_API_KEY> \
>
  --namespace <your-namespace>

Out of Memory During Profiling

Symptoms: OOM errors in profiling jobs

Solutions:

1. Reduce gpu_memory_utilization in engine config
2. Reduce --max-context-length
3. Skip larger TP configurations
4. Use fewer GPUs per test

Unsupported Parallelization Mapping in Backend

Symptoms: Starttime/runtime error in the backend. For example, prime number of attention heads restrain TP size to be 1 (i.e., falcon-7b with 71 attention heads). Or some backend does not support different TP sizes for prefill and decode.

Solutions:

1. Contact the backend to add support for the use cases and bump backend version in dynamo.
2. Restrain the max and min number of GPUs per engine to the supported range.

Next Steps

• Deploy with DGDR: See Quick Start Guide
• Understand SLA Planner: Read SLA Planner Deep Dive
• Monitor Deployments: Set up Observability
• Optimize Performance: See Performance Tuning

Related Documentation

• DGDR API Reference
• SLA Planner Quick Start
• SLA Planner Architecture
• Profiler Arguments Reference