> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Fault Tolerance Overview

Dynamo fault tolerance has two layers:

- **Request fault tolerance** protects the client-visible request path. Use these guides when you need to recover in-flight requests, reject new work under overload, or stop wasted work after client disconnects.
- **Worker fault tolerance** protects serving capacity as workers drain, fail, or recover. Use these guides when you need Kubernetes pods to shut down cleanly, recover engines locally, or understand how Dynamo discovers and routes around worker loss.

Most production deployments need both. Request fault tolerance keeps individual generations from failing unnecessarily, while worker fault tolerance keeps the worker pool stable as Kubernetes reschedules pods or hardware faults occur.

## Request Fault Tolerance

These behaviors operate at the request boundary: an incoming request, an in-flight generation, or a client connection.

- **[Request Migration](/dynamo/dev/kubernetes/fault-tolerance/request-migration)** — Recovers an in-flight generation when a worker fails mid-request by moving the request to another healthy worker. **Off by default** — enable it when you want best-effort continuity for long-running generations.
- **[Request Rejection](/dynamo/dev/kubernetes/fault-tolerance/request-rejection)** — Rejects new requests with HTTP 529 when every worker is too busy, so clients can retry instead of adding queueing delay for everyone. **Off by default** — enable it when you want explicit overload behavior.
- **[Request Cancellation](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture)** — Stops frontend and runtime work when the client disconnects. This is a built-in runtime behavior and does not require workload configuration.

## Worker Fault Tolerance

These behaviors operate at the worker and engine lifecycle boundary: planned shutdown, pod failure, engine failure, and service discovery.

- **[Graceful Shutdown](/dynamo/dev/kubernetes/fault-tolerance/graceful-shutdown)** — Lets a worker finish the requests it is already handling before Kubernetes terminates the pod. **On by default** — tune the grace period to match your rollout and scale-down behavior.
- **[Shadow Engine Failover](/dynamo/dev/kubernetes/fault-tolerance/shadow-engine-failover)** — Runs an active/passive engine pair on the same node so a shadow engine can take over locally after an engine failure. It does not preserve in-flight requests or KV cache state.
- **[Health Check Reference](/dynamo/dev/reference/observability/health-checks)** — Documents the liveness, readiness, and engine-monitoring endpoints used to detect unhealthy workers.
- **[Distributed Runtime](/dynamo/dev/knowledge-base/concepts/system-architecture/distributed-runtime)** — Explains the service discovery and lease mechanisms Dynamo uses to detect worker loss and route new traffic to healthy capacity.

## Testing and References

Use these Knowledge Base pages when you want the deeper implementation model or validation details:

- [Fault Tolerance Testing](/dynamo/dev/knowledge-base/concepts/fault-tolerance/fault-tolerance-testing) — the framework for validating these behaviors (cancellation, migration, etcd HA failover, hardware fault injection).
- [Request Migration Architecture](/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-migration-architecture) — pipeline position, token-state tracking, and worker-failure scenarios.

## Configuration Reference

Every flag and environment variable for configurable fault tolerance behavior is cataloged in the Reference tab:

- [Frontend Configuration](/dynamo/dev/reference/components/frontend-configuration) — migration limits, independent busy thresholds, overload status, and the threshold API.
- [Runtime Configuration](/dynamo/dev/reference/components/runtime-configuration) — local worker inhibition and worker-side engine and queue limits.
- [Observability Environment Variables](/dynamo/dev/reference/observability/environment-variables) — health-check and system-port variables.