--- title: "Router - Load Balancing" description: "import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';" type: skill canonical_url: https://claudary.paisolsolutions.com/skills/routing-2 source: "Claudary" difficulty: intermediate author: "Claude Code Knowledge Pack" date: 2026-07-10T11:46:14.646Z license: CC-BY-4.0 attribution: "Router - Load Balancing โ€” Claudary (https://claudary.paisolsolutions.com/skills/routing-2)" --- # Router - Load Balancing import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ## Overview import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Router - Load Balancing LiteLLM manages: - Load-balance across multiple deployments (e.g. Azure/OpenAI) - Prioritizing important requests to ensure they don't fail (i.e. Queueing) - Basic reliability logic - cooldowns, fallbacks, timeouts and retries (fixed + exponential backoff) across multiple deployments/providers. In production, litellm supports using Redis as a way to track cooldown server and usage (managing tpm/rpm limits). :::info If you want a server to load balance across different LLM APIs, use our [LiteLLM Proxy Server](./proxy/load_balancing.md) ::: ## Load Balancing (s/o [@paulpierre](https://www.linkedin.com/in/paulpierre/) and [sweep proxy](https://docs.sweep.dev/blogs/openai-proxy) for their contributions to this implementation) [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py) ### Quick Start Loadbalance across multiple [azure](./providers/azure)/[bedrock](./providers/bedrock.md)/[provider](./providers/) deployments. LiteLLM will handle retrying in different regions if a call fails. ```python from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-3.5-turbo", # model alias -> loadbalance between models with same `model_name` "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") } }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") } }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo", "api_key": os.getenv("OPENAI_API_KEY"), } }, { "model_name": "gpt-4", "litellm_params": { # params for litellm completion/embedding call "model": "azure/gpt-4", "api_key": os.getenv("AZURE_API_KEY"), "api_base": os.getenv("AZURE_API_BASE"), "api_version": os.getenv("AZURE_API_VERSION"), } }, { "model_name": "gpt-4", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-4", "api_key": os.getenv("OPENAI_API_KEY"), } }, ] router = Router(model_list=model_list) # openai.ChatCompletion.create replacement # requests with model="gpt-3.5-turbo" will pick a deployment where model_name="gpt-3.5-turbo" response = await router.acompletion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) # openai.ChatCompletion.create replacement # requests with model="gpt-4" will pick a deployment where model_name="gpt-4" response = await router.acompletion(model="gpt-4", messages=[{"role": "user", "content": "Hey, how's it going?"}]) print(response) ``` :::info See detailed proxy loadbalancing/fallback docs [here](./proxy/reliability.md) ::: 1. Setup model_list with multiple deployments ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/ api_base: api_key: - model_name: gpt-3.5-turbo litellm_params: model: azure/gpt-turbo-small-ca api_base: https://my-endpoint-canada-berri992.openai.azure.com/ api_key: - model_name: gpt-3.5-turbo litellm_params: model: azure/gpt-turbo-large api_base: https://openai-france-1234.openai.azure.com/ api_key: ``` 2. Start proxy ```bash litellm --config /path/to/config.yaml ``` 3. Test it! ```bash curl -X POST 'http://0.0.0.0:4000/chat/completions' \\ -H 'Content-Type: application/json' \\ -H 'Authorization: Bearer sk-1234' \\ -d '{ "model": "gpt-3.5-turbo", "messages": [ {"role": "user", "content": "Hi there!"} ], "mock_testing_rate_limit_error": true }' ``` ### Available Endpoints - `router.completion()` - chat completions endpoint to call 100+ LLMs - `router.acompletion()` - async chat completion calls - `router.embedding()` - embedding endpoint for Azure, OpenAI, Huggingface endpoints - `router.aembedding()` - async embeddings calls - `router.text_completion()` - completion calls in the old OpenAI `/v1/completions` endpoint format - `router.atext_completion()` - async text completion calls - `router.image_generation()` - completion calls in OpenAI `/v1/images/generations` endpoint format - `router.aimage_generation()` - async image generation calls ## Advanced - Routing Strategies โญ๏ธ #### Routing Strategies - Weighted Pick, Rate Limit Aware, Least Busy, Latency Based, Cost Based Router provides multiple strategies for routing your calls across multiple deployments. **We recommend using `simple-shuffle` (default) for best performance in production.** **Default and Recommended for Production** - Best performance with minimal latency overhead. Picks a deployment based on the provided **Requests per minute (rpm) or Tokens per minute (tpm)** If `rpm` or `tpm` is not provided, it randomly picks a deployment You can also set a `weight` param, to specify which model should get picked when. ##### **LiteLLM Proxy Config.yaml** ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/chatgpt-v-2 api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE rpm: 900 - model_name: gpt-3.5-turbo litellm_params: model: azure/chatgpt-functioncalling api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE rpm: 10 ``` ##### **Python SDK** ```python from litellm import Router import asyncio model_list = [{ # list of model deployments "model_name": "gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "rpm": 900, # requests per minute for this API } }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "rpm": 10, } },] # init router router = Router(model_list=model_list, routing_strategy="simple-shuffle") async def router_acompletion(): response = await router.acompletion( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) return response asyncio.run(router_acompletion()) ``` ##### **LiteLLM Proxy Config.yaml** ```yaml model_list: - model_name: gpt-3.5-turbo litellm_params: model: azure/chatgpt-v-2 api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE weight: 9 - model_name: gpt-3.5-turbo litellm_params: model: azure/chatgpt-functioncalling api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE weight: 1 ``` ##### **Python SDK** ```python from litellm import Router import asyncio model_list = [{ "model_name": "gpt-3.5-turbo", # model alias "litellm_params": { "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "weight": 9, # pick this 90% of the time } }, { "model_name": "gpt-3.5-turbo", "litellm_params": { "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "weight": 1, } }] # init router router = Router(model_list=model_list, routing_strategy="simple-shuffle") async def router_acompletion(): response = await router.acompletion( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}] ) print(response) return response asyncio.run(router_acompletion()) ``` > [!WARNING] **Usage-based routing is not recommended for production due to performance impacts.** Use `simple-shuffle` (default) for optimal performance in high-traffic scenarios. Usage-based routing adds significant latency due to Redis operations for tracking usage across deployments. **๐ŸŽ‰ NEW** This is an async implementation of usage-based-routing. **Filters out deployment if tpm/rpm limit exceeded** - If you pass in the deployment's tpm/rpm limits. Routes to **deployment with lowest TPM usage** for that minute. In production, we use Redis to track usage (TPM/RPM) across multiple deployments. This implementation uses **async redis calls** (redis.incr and redis.mget). For Azure, [you get 6 RPM per 1000 TPM](https://stackoverflow.com/questions/77368844/what-is-the-request-per-minute-rate-limit-for-azure-openai-models-for-gpt-3-5-tu) ```python from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") "tpm": 100000, "rpm": 10000, }, }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") "tpm": 100000, "rpm": 1000, }, }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo", "api_key": os.getenv("OPENAI_API_KEY"), "tpm": 100000, "rpm": 1000, }, }] router = Router(model_list=model_list, redis_host=os.environ["REDIS_HOST"], redis_password=os.environ["REDIS_PASSWORD"], redis_port=os.environ["REDIS_PORT"], routing_strategy="simple-shuffle" # ๐Ÿ‘ˆ RECOMMENDED - best performance enable_pre_call_checks=True, # enables router rate limits for concurrent calls ) response = await router.acompletion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}] print(response) ``` **1. Set strategy in config** ```yaml model_list: - model_name: gpt-3.5-turbo # model alias litellm_params: # params for litellm completion/embedding call model: azure/chatgpt-v-2 # actual model name api_key: os.environ/AZURE_API_KEY api_version: os.environ/AZURE_API_VERSION api_base: os.environ/AZURE_API_BASE tpm: 100000 rpm: 10000 - model_name: gpt-3.5-turbo litellm_params: # params for litellm completion/embedding call model: gpt-3.5-turbo api_key: os.getenv(OPENAI_API_KEY) tpm: 100000 rpm: 1000 router_settings: routing_strategy: simple-shuffle # ๐Ÿ‘ˆ RECOMMENDED - best performance redis_host: redis_password: redis_port: enable_pre_call_check: true general_settings: master_key: sk-1234 ``` **2. Start proxy** ```bash litellm --config /path/to/config.yaml ``` **3. Test it!** ```bash curl --location 'http://localhost:4000/v1/chat/completions' \\ --header 'Content-Type: application/json' \\ --header 'Authorization: Bearer sk-1234' \\ --data '{ "model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hey, how's it going?"}] }' ``` Picks the deployment with the lowest response time. It caches, and updates the response times for deployments based on when a request was sent and received from a deployment. [**How to test**](https://github.com/BerriAI/litellm/blob/main/tests/local_testing/test_lowest_latency_routing.py) ```python from litellm import Router import asyncio model_list = [{ ... }] # init router router = Router(model_list=model_list, routing_strategy="latency-based-routing",# ๐Ÿ‘ˆ set routing strategy enable_pre_call_check=True, # enables router rate limits for concurrent calls ) ## CALL 1+2 tasks = [] response = None final_response = None for _ in range(2): tasks.append(router.acompletion(model=model, messages=messages)) response = await asyncio.gather(*tasks) if response is not None: ## CALL 3 await asyncio.sleep(1) # let the cache update happen picked_deployment = router.lowestlatency_logger.get_available_deployments( model_group=model, healthy_deployments=router.healthy_deployments ) final_response = await router.acompletion(model=model, messages=messages) print(f"min deployment id: {picked_deployment}") print(f"model id: {final_response._hidden_params['model_id']}") assert ( final_response._hidden_params["model_id"] == picked_deployment["model_info"]["id"] ) ``` #### Set Time Window Set time window for how far back to consider when averaging latency for a deployment. **In Router** ```python router = Router(..., routing_strategy_args={"ttl": 10}) ``` **In Proxy** ```yaml router_settings: routing_strategy_args: {"ttl": 10} ``` #### Set Lowest Latency Buffer Set a buffer within which deployments are candidates for making calls to. E.g. if you have 5 deployments ``` https://litellm-prod-1.openai.azure.com/: 0.07s https://litellm-prod-2.openai.azure.com/: 0.1s https://litellm-prod-3.openai.azure.com/: 0.1s https://litellm-prod-4.openai.azure.com/: 0.1s https://litellm-prod-5.openai.azure.com/: 4.66s ``` to prevent initially overloading `prod-1`, with all requests - we can set a buffer of 50%, to consider deployments `prod-2, prod-3, prod-4`. **In Router** ```python router = Router(..., routing_strategy_args={"lowest_latency_buffer": 0.5}) ``` **In Proxy** ```yaml router_settings: routing_strategy_args: {"lowest_latency_buffer": 0.5} ```