Proxy - Load Balancing

Load balance multiple instances of the same model

The proxy will handle routing requests (using LiteLLM's Router). Set rpm in the config if you want maximize throughput

:::info

For more details on routing strategies / params, see Routing

:::

How Load Balancing Works

LiteLLM automatically distributes requests across multiple deployments of the same model using its built-in router. the proxy routes traffic to optimize performance and reliability.

"simple-shuffle" routing strategy is used by default

Routing Strategies

:::tip Deployment Priority Use the order parameter to prioritize specific deployments. See Deployment Ordering for details. :::

Quick Start - Load Balancing

Step 1 - Set deployments on config

Example config below. Here requests with model=gpt-3.5-turbo will be routed across multiple instances of azure/gpt-3.5-turbo

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: <your-azure-endpoint>
      api_key: <your-azure-api-key>
      rpm: 6      # Rate limit for this deployment: in requests per minute (rpm)
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-ca
      api_base: https://my-endpoint-canada-berri992.openai.azure.com/
      api_key: <your-azure-api-key>
      rpm: 6
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-large
      api_base: https://openai-france-1234.openai.azure.com/
      api_key: <your-azure-api-key>
      rpm: 1440

router_settings:
  routing_strategy: simple-shuffle # Literal["simple-shuffle", "least-busy", "usage-based-routing","latency-based-routing"], default="simple-shuffle"
  model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models with `gpt-3.5-turbo`
  num_retries: 2
  timeout: 30                                  # 30 seconds
  redis_host: <your redis host>                # set this when using multiple litellm proxy deployments, load balancing state stored in redis
  redis_password: <your redis password>
  redis_port: 1992

Enforce Model Rate Limits

Strictly enforce RPM/TPM limits set on deployments. When limits are exceeded, requests are blocked before reaching the LLM provider with a 429 Too Many Requests error.

:::info By default, rpm and tpm values are only used for routing decisions (picking deployments with capacity). With enforce_model_rate_limits, they become hard limits. :::

Quick Start

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY
    rpm: 60     # 60 requests per minute
    tpm: 90000  # 90k tokens per minute

router_settings:
  optional_pre_call_checks:
    - enforce_model_rate_limits  # 👈 Enables strict enforcement

How It Works

Why TPM is best-effort: Token count is unknown until the LLM responds. TPM is checked before each request (blocks if already over), and tracked after (adds actual tokens used).

Error Response

{
  "error": {
    "message": "Model rate limit exceeded. RPM limit=60, current usage=60",
    "type": "rate_limit_error",
    "code": 429
  }
}

Response includes retry-after: 60 header.

Multi-Instance Deployment

For multiple LiteLLM proxy instances, add Redis to share rate limit state:

router_settings:
  optional_pre_call_checks:
    - enforce_model_rate_limits
  redis_host: redis.example.com
  redis_port: 6379
  redis_password: your-password

:::info Detailed information about routing strategies can be found here :::

Step 2: Start Proxy with config

$ litellm --config /path/to/config.yaml

Test - Simple Call

Here requests with model=gpt-3.5-turbo will be routed across multiple instances of azure/gpt-3.5-turbo

👉 Key Change: model="gpt-3.5-turbo"

Check the model_id in Response Headers to make sure the requests are being load balanced

client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ]
)

print(response)

curl --location 'http://0.0.0.0:4000/chat/completions' \\
    --header 'Content-Type: application/json' \\
    --data '{
    "model": "gpt-3.5-turbo",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
}'

Test - Loadbalancing

In this request, the following will occur:

1. A rate limit exception will be raised
2. LiteLLM proxy will retry the request on the model group (default retries are 3).

curl -X POST 'http://0.0.0.0:4000/chat/completions' \\
-H 'Content-Type: application/json' \\
-H 'Authorization: Bearer sk-1234' \\
-d '{
  "model": "gpt-3.5-turbo",
  "messages": [
        {"role": "user", "content": "Hi there!"}
    ],
    "mock_testing_rate_limit_error": true
}'

See Code

Load Balancing using multiple litellm instances (Kubernetes, Auto Scaling)

LiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass redis_host, redis_password and redis_port to enable this. (LiteLLM will use Redis to track rpm/tpm usage )

Example config

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: <your-azure-endpoint>
      api_key: <your-azure-api-key>
      rpm: 6      # Rate limit for this deployment: in requests per minute (rpm)
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-ca
      api_base: https://my-endpoint-canada-berri992.openai.azure.com/
      api_key: <your-azure-api-key>
      rpm: 6
router_settings:
  redis_host: <your redis host>
  redis_password: <your redis password>
  redis_port: 1992
  cache_params:
    type: redis
    max_connections: 100  # maximum Redis connections in the pool; tune based on expected concurrency/load

Router settings on config - routing_strategy, model_group_alias

Expose an 'alias' for a 'model_name' on the proxy server.

model_group_alias: {
  "gpt-4": "gpt-3.5-turbo"
}

These aliases are shown on /v1/models, /v1/model/info, and /v1/model_group/info by default.

litellm.Router() settings can be set under router_settings. You can set model_group_alias, routing_strategy, num_retries,timeout . See all Router supported params here

Usage

Example config with router_settings

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: <your-azure-endpoint>
      api_key: <your-azure-api-key>

router_settings:
  model_group_alias: {"gpt-4": "gpt-3.5-turbo"} # all requests with `gpt-4` will be routed to models 

Hide Alias Models

Use this if you want to set-up aliases for:

1. typos
2. minor model version changes
3. case sensitive changes between updates

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: <your-azure-endpoint>
      api_key: <your-azure-api-key>

router_settings:
  model_group_alias:
    "GPT-3.5-turbo": # alias
      model: "gpt-3.5-turbo"  # Actual model name in 'model_list'
      hidden: true             # Exclude from `/v1/models`, `/v1/model/info`, `/v1/model_group/info`

Complete Spec

model_group_alias: Optional[Dict[str, Union[str, RouterModelGroupAliasItem]]] = {}

class RouterModelGroupAliasItem(TypedDict):
    model: str
    hidden: bool  # if 'True', don't return on `/v1/models`, `/v1/model/info`, `/v1/model_group/info`

Deployment Ordering (Priority)

Set order in litellm_params to prioritize deployments. Lower values = higher priority. When multiple deployments share the same order, the routing strategy picks among them.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-primary
      api_key: os.environ/AZURE_API_KEY
      order: 1  # 👈 Highest priority - always tried first

  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-fallback
      api_key: os.environ/AZURE_API_KEY_2
      order: 2  # 👈 Used when order=1 fails

How order-based fallback works

When a request to an order=1 deployment fails (connection error, 404, 429, etc.), the router automatically tries order=2 deployments, then order=3, and so on. Each order level gets its own set of retries before escalating to the next.

If all order levels are exhausted, the router falls through to any configured model-level fallbacks.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-primary
      api_key: os.environ/AZURE_API_KEY
      order: 1

  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-secondary
      api_key: os.environ/AZURE_API_KEY_2
      order: 2

  - model_name: gpt-4-fallback
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  fallbacks:
    - gpt-4:
        - gpt-4-fallback  # tried after all order levels fail

The fallback chain for the above config: order=1 → order=2 → gpt-4-fallback.

For 429 (rate limit) errors specifically, the failed deployment is immediately placed on cooldown. If all order=1 deployments are on cooldown, the router picks order=2 deployments directly during retries without waiting for the fallback path.

Team-scoped models and legacy model_aliases {#team-scoped-models-and-legacy-model_aliases}

Team-scoped deployments are identified by model_info.team_id and model_info.team_public_model_name. Requests should use the public model name; the router resolves all sibling deployments (same public name, different api_base / order, etc.) for routing, failover, and deployment order.

For router internals: when a team_id is in scope, optimized lookups key off (team_id, team_public_model_name). If code passes an internal deployment id (e.g. model_name_<team_id>_<uuid>) instead of the public name, routing still works via the usual deployment-name paths, but the team-specific fast path applies only to the public name.

Legacy teams: Older proxy versions could persist model_aliases on the team row mapping a public name to a single internal deployment id (model_name_<team_id>_<uuid>). On each request, pre-call logic may still rewrite model to that internal name before routing, which collapses to one deployment and can make newer sibling deployments unreachable.

Migration options:

1. Recommended for upgrades: Set environment variable LITELLM_ENABLE_TEAM_STALE_ALIAS_BYPASS=true so that when sibling team deployments exist for the public name, the stale alias rewrite is skipped and team-scoped routing (including order and failover) applies. See the Environment variables table in the proxy settings doc.
2. Data cleanup: Remove obsolete model_aliases entries for team public names from the team record in the database so only team_public_model_name + team model list drive access.

If a stale alias is detected and the bypass is not enabled, the proxy may emit a one-time warning in logs explaining that sibling deployments may be unreachable until the flag is set or aliases are cleaned up.

When You'll See Load Balancing in Action

Immediate Effects:

• Different deployments serve subsequent requests (visible in logs)
• Better response times during high traffic

Observable Benefits:

• Higher throughput: More requests handled simultaneously across deployments
• Improved reliability: If one deployment fails, traffic automatically routes to healthy ones
• Better resource utilization: Load spread evenly across all available deployments

Special Considerations for Responses API

When load balancing OpenAI's Responses API across deployments with different API keys (e.g., different Azure regions or organizations), encrypted content items (like rs_... reasoning items) can only be decrypted by the originating API key.

Solution: Use the encrypted_content_affinity pre-call check (requires LiteLLM >= 1.82.3) to automatically route follow-up requests containing encrypted items to the correct deployment:

model_list:
  - model_name: gpt-5.1-codex
    litellm_params:
      model: azure/gpt-5.1-codex
      api_base: https://eastus.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_EASTUS
    model_info:
      id: "deployment-eastus"
  
  - model_name: gpt-5.1-codex
    litellm_params:
      model: azure/gpt-5.1-codex
      api_base: https://westeurope.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY_WESTEUROPE
    model_info:
      id: "deployment-westeurope"

router_settings:
  optional_pre_call_checks:
    - encrypted_content_affinity  # 👈 Prevents invalid_encrypted_content errors

This ensures requests containing encrypted content are routed to the deployment that created them, while other requests continue to load balance normally.

Learn more about Encrypted Content Affinity →