--- title: "Health Check Driven Routing" description: "Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from the routing pool proactively, not after a user request already failed." type: skill canonical_url: https://claudary.paisolsolutions.com/skills/health-check-routing source: "Claudary" difficulty: intermediate author: "Claude Code Knowledge Pack" date: 2026-07-10T11:25:09.937Z license: CC-BY-4.0 attribution: "Health Check Driven Routing — Claudary (https://claudary.paisolsolutions.com/skills/health-check-routing)" --- # Health Check Driven Routing Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from the routing pool proactively, not after a user request already failed. ## Overview # Health Check Driven Routing Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from the routing pool proactively, not after a user request already failed. ## Architecture ## What problem does this solve? By default, LiteLLM routes traffic to all deployments and only stops sending to a broken one after it has already failed a user request. The cooldown system is reactive. Health check driven routing makes this **proactive**: a background loop pings every deployment on a configurable interval. If a deployment fails its health check, it gets removed from the routing pool immediately, before a user request lands on it. When you also set `allowed_fails_policy`, you control exactly how many health check failures of each error type (auth errors, rate limits, timeouts) are needed before a deployment enters cooldown. This avoids false positives from transient noise. ## Setup ### Step 1: Enable background health checks Background health checks are off by default. Turn them on in `general_settings`: ```yaml general_settings: background_health_checks: true health_check_interval: 60 # seconds between each full check cycle ``` ### Step 2: Enable health check routing ```yaml general_settings: background_health_checks: true health_check_interval: 60 enable_health_check_routing: true # ← route away from unhealthy deployments ``` At this point, any deployment that fails its health check is immediately excluded from routing until the next check cycle clears it. ### Step 3: Add a policy to control how many failures trigger cooldown Without a policy, the first health check failure marks a deployment as unhealthy. If you want more tolerance (e.g., only act after 2 consecutive auth failures), use `allowed_fails_policy`: ```yaml model_list: - model_name: claude-sonnet litellm_params: model: anthropic/claude-sonnet-4-5 api_key: os.environ/ANTHROPIC_API_KEY - model_name: claude-sonnet litellm_params: model: anthropic/claude-sonnet-4-5 api_key: os.environ/ANTHROPIC_API_KEY_SECONDARY general_settings: background_health_checks: true health_check_interval: 30 enable_health_check_routing: true router_settings: cooldown_time: 60 # how long a deployment stays in cooldown allowed_fails_policy: AuthenticationErrorAllowedFails: 1 # cooldown after 2nd auth failure TimeoutErrorAllowedFails: 3 # cooldown after 4th timeout ``` When `allowed_fails_policy` is set, the binary health check filter is bypassed. Only the cooldown system controls routing exclusion, and it only fires after your configured threshold is crossed. ### Step 4 (optional): Ignore transient errors 429 (rate limit) and 408 (timeout) from a health check usually mean the deployment is temporarily overloaded, not broken. To prevent these from affecting routing at all: ```yaml general_settings: background_health_checks: true health_check_interval: 30 enable_health_check_routing: true health_check_ignore_transient_errors: true # 429 and 408 never affect routing ``` With this on, only hard failures (401, 404, 5xx) from health checks contribute to cooldown. ## Full example ```yaml model_list: - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY_SECONDARY - model_name: gpt-4o litellm_params: model: azure/gpt-4o api_base: os.environ/AZURE_API_BASE api_key: os.environ/AZURE_API_KEY general_settings: background_health_checks: true health_check_interval: 30 enable_health_check_routing: true health_check_ignore_transient_errors: true router_settings: cooldown_time: 60 allowed_fails_policy: AuthenticationErrorAllowedFails: 0 # cooldown immediately on auth failure TimeoutErrorAllowedFails: 2 # cooldown after 3 timeouts RateLimitErrorAllowedFails: 5 # cooldown after 6 rate limits (if not ignoring transients) ``` ## Configuration reference | Setting | Where | Default | Description | |---|---|---|---| | `enable_health_check_routing` | `general_settings` | `false` | Route away from deployments that fail health checks | | `background_health_checks` | `general_settings` | `false` | Must be `true` for health check routing to work | | `health_check_interval` | `general_settings` | `300` | Seconds between full health check cycles | | `health_check_staleness_threshold` | `general_settings` | `interval x 2` | Seconds before cached health state is ignored | | `health_check_ignore_transient_errors` | `general_settings` | `false` | Ignore 429 and 408 from health checks; these never affect routing | | `cooldown_time` | `router_settings` | `5` | Seconds a deployment stays in cooldown after threshold is crossed | | `allowed_fails_policy` | `router_settings` | `null` | Per-error-type failure thresholds before cooldown (see below) | ### `allowed_fails_policy` fields | Field | Error type | HTTP status | |---|---|---| | `AuthenticationErrorAllowedFails` | Bad API key | 401 | | `TimeoutErrorAllowedFails` | Request timeout | 408 | | `RateLimitErrorAllowedFails` | Rate limit exceeded --- Source: [Claudary](https://claudary.paisolsolutions.com/skills/health-check-routing) · https://claudary.paisolsolutions.com