--- title: "Proxy - Load Balancing" description: "import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';" type: skill canonical_url: https://claudary.paisolsolutions.com/skills/load-balancing source: "Claudary" difficulty: intermediate author: "Claude Code Knowledge Pack" date: 2026-07-10T11:30:39.342Z license: CC-BY-4.0 attribution: "Proxy - Load Balancing โ€” Claudary (https://claudary.paisolsolutions.com/skills/load-balancing)" --- # Proxy - Load Balancing import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Overview import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Proxy - Load Balancing Load balance multiple instances of the same model The proxy will handle routing requests (using LiteLLM's Router). **Set `rpm` in the config if you want maximize throughput** :::info For more details on routing strategies / params, see [Routing](../routing.md) ::: ## How Load Balancing Works LiteLLM automatically distributes requests across multiple deployments of the same model using its built-in router. the proxy routes traffic to optimize performance and reliability. "simple-shuffle" routing strategy is used by default ### Routing Strategies | Strategy | Description | When to Use | |----------|-------------|-------------| | **simple-shuffle** (recommended) | Randomly distributes requests | General purpose, good for even load distribution | | **least-busy** | Routes to deployment with fewest active requests | High concurrency scenarios | | **usage-based-routing** (bad for perf) | Routes to deployment with lowest current usage (RPM/TPM) | When you want to respect rate limits evenly | | **latency-based-routing** | Routes to fastest responding deployment | Latency-critical applications | | **cost-based-routing** | Routes to deployment with lowest cost | Cost-sensitive applications | :::tip Deployment Priority Use the `order` parameter to prioritize specific deployments. [See Deployment Ordering](#deployment-ordering-priority) for details. ::: ## Quick Start - Load Balancing #### Step 1 - Set deployments on config **Example config below**. Here requests with `model=gpt-3.5-turbo` will be routed across multiple instances of `azure/gpt-3.5-turbo` ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/ api_base: api_key: rpm: 6 # Rate limit for this deployment: in requests per minute (rpm) - model_name: gpt-3.5-turbo litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: rpm: 6 - model_name: gpt-3.5-turbo litellm_params: model: azure/gpt-turbo-large api_base: https://openai-france-1234.openai.azure.com/ api_key: rpm: 1440 router_settings: routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle" model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models with `gpt-3.5-turbo` num_retries: 2 timeout: 30 # 30 seconds redis_host: # set this when using multiple litellm proxy deployments, load balancing state stored in redis redis_password: redis_port: 1992 ``` ## Enforce Model Rate Limits Strictly enforce RPM/TPM limits set on deployments. When limits are exceeded, requests are blocked **before** reaching the LLM provider with a `429 Too Many Requests` error. :::info By default, `rpm` and `tpm` values are only used for **routing decisions** (picking deployments with capacity). With `enforce_model_rate_limits`, they become **hard limits**. ::: ### Quick Start ```yaml model_list: - model_name: gpt-4 litellm_params: model: openai/gpt-4 api_key: os.environ/OPENAI_API_KEY rpm: 60 # 60 requests per minute tpm: 90000 # 90k tokens per minute router_settings: optional_pre_call_checks: - enforce_model_rate_limits # ๐Ÿ‘ˆ Enables strict enforcement ``` ### How It Works | Limit Type | Enforcement | Accuracy | |------------|-------------|----------| | **RPM** | Hard limit - blocked at exact threshold | 100% accurate | | **TPM** | Best-effort - may slightly exceed | Blocked when already over limit | **Why TPM is best-effort:** Token count is unknown until the LLM responds. TPM is checked before each request (blocks if already over), and tracked after (adds actual tokens used). ### Error Response ```json { "error": { "message": "Model rate limit exceeded. RPM limit=60, current usage=60", "type": "rate_limit_error", "code": 429 } } ``` Response includes `retry-after: 60` header. ### Multi-Instance Deployment For multiple LiteLLM proxy instances, add Redis to share rate limit state: ```yaml router_settings: optional_pre_call_checks: - enforce_model_rate_limits redis_host: redis.example.com redis_port: 6379 redis_password: your-password ``` :::info Detailed information about [routing strategies can be found here](../routing) ::: #### Step 2: Start Proxy with config ```shell $ litellm --config /path/to/config.yaml ``` ### Test - Simple Call Here requests with model=gpt-3.5-turbo will be routed across multiple instances of azure/gpt-3.5-turbo ๐Ÿ‘‰ Key Change: `model="gpt-3.5-turbo"` **Check the `model_id` in Response Headers to make sure the requests are being load balanced** ```python import openai client = openai.OpenAI( api_key="anything", base_url="http://0.0.0.0:4000" ) response = client.chat.completions.create( model="gpt-3.5-turbo", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ] ) print(response) ``` ```shell curl --location 'http://0.0.0.0:4000/chat/completions' \\ --header 'Content-Type: application/json' \\ --data '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "what llm are you" } ] }' ``` ### Test - Loadbalancing In this request, the following will occur: 1. A rate limit exception will be raised 2. LiteLLM proxy will retry the request on the model group (default retries are 3). ```bash curl -X POST 'http://0.0.0.0:4000/chat/completions' \\ -H 'Content-Type: application/json' \\ -H 'Authorization: Bearer sk-1234' \\ -d '{ "model": "gpt-3.5-turbo", "messages": [ {"role": "user", "content": "Hi there!"} ], "mock_testing_rate_limit_error": true }' ``` [**See Code**](https://github.com/BerriAI/litellm/blob/6b8806b45f970cb2446654d2c379f8dcaa93ce3c/litellm/router.py#L2535) ## Load Balancing using multiple litellm instances (Kubernetes, Auto Scaling) LiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass `redis_host`, `redis_password` and `redis_port` to enable this. (LiteLLM will use Redis to track rpm/tpm usage ) Example config ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/ api_base: api_key: rpm: 6 # Rate limit for this deployment: in requests per minute (rpm) - model_name: gpt-3.5-turbo litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: rpm: 6 router_settings: redis_host: redis_password: redis_port: 1992 cache_params: type: redis max_connections: 100 # maximum Redis connections in the pool; tune based on expected concurrency/load ``` ## Router settings on config - routing_strategy, model_group_alias Expose an 'alias' for a 'model_name' on the proxy server. ``` model_group_alias: { "gpt-4": "gpt-3.5-turbo" } ``` These aliases are shown on `/v1/models`, `/v1/model/info`, and `/v1/model_group/info` by default. litellm.Router() settings can be set under `router_settings`. You can set `model_group_alias`, `routing_strategy`, `num_retries`,`timeout` . See all Router supported params [here](https://github.com/BerriAI/litellm/blob/1b942568897a48f014fa44618ec3ce54d7570a46/litellm/router.py#L64) ### Usage Example config with `router_settings` ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/ api_base: api_key: router_settings: model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models ``` ### Hide Alias Models Use this if you want to set-up aliases for: 1. typos 2. minor model version changes 3. case sensitive changes between updates ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/ api_base: api_key: router_settings: model_group_alias: "GPT-3.5-turbo": # alias model: "gpt-3.5-turbo" # Actual model name in 'model_list' hidden: true # Exclude from `/v1/models`, `/v1/model/info`, `/v1/model_group/info` ``` ### Complete Spec ```python model_group_alias: Optional[Dict[str, Union[str, RouterModelGroupAliasItem]]] = {} class RouterModelGroupAliasItem(TypedDict): model: str hidden: bool # if 'True', don't return on `/v1/models`, `/v1/model/info`, `/v1/model_group/info` ``` ## Deployment Ordering (Priority) Set `order` in `litellm_params` to prioritize deployments. Lower values = higher priority. When multiple deployments share the same `order`, the routing strategy picks among them. ```yaml model_list: - model_name: gpt-4 litellm_params: model: azure/gpt-4-primary api_key: os.environ/AZURE_API_KEY order: 1 # ๐Ÿ‘ˆ Highest priority - always tried first - model_name: gpt-4 litellm_params: model: azure/gpt-4-fallback api_key: os.environ/AZURE_API_KEY_2 order: 2 # ๐Ÿ‘ˆ Used when order=1 fails ``` ### How order-based fallback works When a request to an `order=1` deployment fails (connection error, 404, 429, etc.), the router automatically tries `order=2` deployments, then `order=3`, and so on. Each order level gets its own set of retries before escalating to the next. If all order levels are exhausted, the router falls through to any configured [model-level fallbacks](#fallbacks). ```yaml model_list: - model_name: gpt-4 litellm_params: model: azure/gpt-4-primary api_key: os.environ/AZURE_API_KEY order: 1 - model_name: gpt-4 litellm_params: model: azure/gpt-4-secondary api_key: os.environ/AZURE_API_KEY_2 order: 2 - model_name: gpt-4-fallback litellm_params: model: openai/gpt-4 api_key: os.environ/OPENAI_API_KEY router_settings: fallbacks: - gpt-4: - gpt-4-fallback # tried after all order levels fail ``` The fallback chain for the above config: `order=1` โ†’ `order=2` โ†’ `gpt-4-fallback`. For 429 (rate limit) errors specifically, the failed deployment is immediately placed on cooldown. If all `order=1` deployments are on cooldown, the router picks `order=2` deployments directly during retries without waiting for the fallback path. ### Team-scoped models and legacy `model_aliases` {#team-scoped-models-and-legacy-model_aliases} Team-scoped deployments are identified by `model_info.team_id` and `model_info.team_public_model_name`. Requests should use the **public** model name; the router resolves all sibling deployments (same public name, different `api_base` / `order`, etc.) for routing, failover, and deployment `order`. For router internals: when a `team_id` is in scope, optimized lookups key off `(team_id, team_public_model_name)`. If code passes an internal deployment id (e.g. `model_name__`) instead of the public name, routing still works via the usual deployment-name paths, but the team-specific fast path applies only to the public name. **Legacy teams:** Older proxy versions could persist `model_aliases` on the team row mapping a public name to a single internal deployment id (`model_name__`). On each request, pre-call logic may still rewrite `model` to that internal name **before** routing, which collapses to one deployment and can make newer sibling deployments unreachable. **Migration options:** 1. **Recommended for upgrades:** Set environment variable `LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS=true` so that when sibling team deployments exist for the public name, the stale alias rewrite is skipped and team-scoped routing (including `order` and failover) applies. See the [Environment variables](./config_settings) table in the proxy settings doc. 2. **Data cleanup:** Remove obsolete `model_aliases` entries for team public names from the team record in the database so only `team_public_model_name` + team model list drive access. If a stale alias is detected and the bypass is **not** enabled, the proxy may emit a **one-time** warning in logs explaining that sibling deployments may be unreachable until the flag is set or aliases are cleaned up. ### When You'll See Load Balancing in Action **Immediate Effects:** - Different deployments serve subsequent requests (visible in logs) - Better response times during high traffic **Observable Benefits:** - **Higher throughput**: More requests handled simultaneously across deployments - **Improved reliability**: If one deployment fails, traffic automatically routes to healthy ones - **Better resource utilization**: Load spread evenly across all available deployments ## Special Considerations for Responses API When load balancing OpenAI's Responses API across deployments with **different API keys** (e.g., different Azure regions or organizations), encrypted content items (like `rs_...` reasoning items) can only be decrypted by the originating API key. **Solution:** Use the `encrypted_content_affinity` pre-call check (requires LiteLLM >= 1.82.3) to automatically route follow-up requests containing encrypted items to the correct deployment: ```yaml model_list: - model_name: gpt-5.1-codex litellm_params: model: azure/gpt-5.1-codex api_base: https://eastus.openai.azure.com/ api_key: os.environ/AZURE_API_KEY_EASTUS model_info: id: "deployment-eastus" - model_name: gpt-5.1-codex litellm_params: model: azure/gpt-5.1-codex api_base: https://westeurope.openai.azure.com/ api_key: os.environ/AZURE_API_KEY_WESTEUROPE model_info: id: "deployment-westeurope" router_settings: optional_pre_call_checks: - encrypted_content_affinity # ๐Ÿ‘ˆ Prevents invalid_encrypted_content errors ``` This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally. **[Learn more about Encrypted Content Affinity โ†’](../response_api.md#encrypted-content-affinity-multi-region-load-balancing)** --- Source: [Claudary](https://claudary.paisolsolutions.com/skills/load-balancing) ยท https://claudary.paisolsolutions.com