Skillintermediate

Caching

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Claude Code Knowledge Pack7/10/2026

Overview

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Caching

:::note

For OpenAI/Anthropic Prompt Caching, go here

:::

Cache LLM Responses. LiteLLM's caching system stores and reuses LLM responses to save costs and reduce latency. When you make the same request twice, the cached response is returned instead of calling the LLM API again.

Supported Caches

In Memory Cache
Disk Cache
Redis Cache
Qdrant Semantic Cache
Redis Semantic Cache
S3 Bucket Cache
GCS Bucket Cache

Quick Start

Caching can be enabled by adding the cache key in the config.yaml

Step 1: Add `cache` to the config.yaml

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
  - model_name: text-embedding-ada-002
    litellm_params:
      model: text-embedding-ada-002

litellm_settings:
  set_verbose: True
  cache: True # set cache responses to True, litellm defaults to using a redis cache

[OPTIONAL] Step 1.5: Add redis namespaces, default ttl

Namespace

If you want to create some folder for your keys, you can set a namespace, like this:

litellm_settings:
  cache: true
  cache_params: # set cache params for redis
    type: redis
    namespace: "litellm.caching.caching"

and keys will be stored like:

litellm.caching.caching:<hash>

Redis Cluster

model_list:
  - model_name: "*"
    litellm_params:
      model: "*"

litellm_settings:
  cache: True
  cache_params:
    type: redis
    redis_startup_nodes: [{ "host": "127.0.0.1", "port": "7001" }]

You can configure redis cluster in your .env by setting REDIS_CLUSTER_NODES in your .env

Example REDIS_CLUSTER_NODES value

REDIS_CLUSTER_NODES = "[{"host": "127.0.0.1", "port": "7001"}, {"host": "127.0.0.1", "port": "7003"}, {"host": "127.0.0.1", "port": "7004"}, {"host": "127.0.0.1", "port": "7005"}, {"host": "127.0.0.1", "port": "7006"}, {"host": "127.0.0.1", "port": "7007"}]"

:::note

Example python script for setting redis cluster nodes in .env:

# List of startup nodes
startup_nodes = [
    {"host": "127.0.0.1", "port": "7001"},
    {"host": "127.0.0.1", "port": "7003"},
    {"host": "127.0.0.1", "port": "7004"},
    {"host": "127.0.0.1", "port": "7005"},
    {"host": "127.0.0.1", "port": "7006"},
    {"host": "127.0.0.1", "port": "7007"},
]

# set startup nodes in environment variables
os.environ["REDIS_CLUSTER_NODES"] = json.dumps(startup_nodes)
print("REDIS_CLUSTER_NODES", os.environ["REDIS_CLUSTER_NODES"])

:::

Redis Sentinel

model_list:
  - model_name: "*"
    litellm_params:
      model: "*"

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    service_name: "mymaster"
    sentinel_nodes: [["localhost", 26379]]
    sentinel_password: "password" # [OPTIONAL]

You can configure redis sentinel in your .env by setting REDIS_SENTINEL_NODES in your .env

Example REDIS_SENTINEL_NODES value

REDIS_SENTINEL_NODES='[["localhost", 26379]]'
REDIS_SERVICE_NAME = "mymaster"
REDIS_SENTINEL_PASSWORD = "password"

:::note

Example python script for setting redis cluster nodes in .env:

# List of startup nodes
sentinel_nodes = [["localhost", 26379]]

# set startup nodes in environment variables
os.environ["REDIS_SENTINEL_NODES"] = json.dumps(sentinel_nodes)
print("REDIS_SENTINEL_NODES", os.environ["REDIS_SENTINEL_NODES"])

:::

TTL

litellm_settings:
  cache: true
  cache_params: # set cache params for redis
    type: redis
    ttl: 600 # will be cached on redis for 600s
    # default_in_memory_ttl: Optional[float], default is None. time in seconds.
    # default_in_redis_ttl: Optional[float], default is None. time in seconds.

SSL

just set REDIS_SSL="True" in your .env, and LiteLLM will pick this up.

REDIS_SSL="True"

For quick testing, you can also use REDIS_URL, eg.:

REDIS_URL="rediss://.."

but we don't recommend using REDIS_URL in prod. We've noticed a performance difference between using it vs. redis_host, port, etc.

GCP IAM Authentication

For GCP Memorystore Redis with IAM authentication, install the required dependency:

:::info IAM authentication for redis is only supported via GCP and only on Redis Clusters for now. :::

uv add google-cloud-iam

For Redis Cluster with GCP IAM:

litellm_settings:
  cache: True
  cache_params:
    type: redis
    redis_startup_nodes:
      [{ "host": "10.128.0.2", "port": 6379 }, { "host": "10.128.0.2", "port": 11008 }]
    gcp_service_account: "projects/-/serviceAccounts/your-sa@project.iam.gserviceaccount.com"
    ssl: true
    ssl_cert_reqs: null
    ssl_check_hostname: false

You can configure GCP IAM Redis authentication in your .env:

For Redis Cluster:

REDIS_CLUSTER_NODES='[{"host": "10.128.0.2", "port": 6379}, {"host": "10.128.0.2", "port": 11008}]'
REDIS_GCP_SERVICE_ACCOUNT="projects/-/serviceAccounts/your-sa@project.iam.gserviceaccount.com"
REDIS_GCP_SSL_CA_CERTS="./server-ca.pem"
REDIS_SSL="True"
REDIS_SSL_CERT_REQS="None"
REDIS_SSL_CHECK_HOSTNAME="False"

GCP Authentication Setup

Make sure your GCP credentials are configured:

# Option 1: Service account key file

# Option 2: If running on GCP compute instance with service account attached
# No additional setup needed

Step 2: Add Redis Credentials to .env

Set either REDIS_URL or the REDIS_HOST in your os environment, to enable caching.

REDIS_URL = ""        # REDIS_URL='redis://username:password@hostname:port/database'
## OR ## 
REDIS_HOST = ""       # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
REDIS_PORT = ""       # REDIS_PORT='18841'
REDIS_PASSWORD = ""   # REDIS_PASSWORD='liteLlmIsAmazing'
REDIS_USERNAME = ""   # REDIS_USERNAME='my-redis-username' [OPTIONAL] if your redis server requires a username
REDIS_SSL = "True"    # REDIS_SSL='True' to enable SSL by default is False

Additional kwargs
:::info Use REDIS_* environment variables to configure all Redis client library parameters. This is the suggested mechanism for toggling Redis settings as it automatically maps environment variables to Redis client kwargs. :::

You can pass in any additional redis.Redis arg, by storing the variable + value in your os environment, like this:

REDIS_<redis-kwarg-name> = ""

For example:

REDIS_SSL = "True"
REDIS_SSL_CERT_REQS = "None" 
REDIS_CONNECTION_POOL_KWARGS = '{"max_connections": 20}'

:::warning Note: For non-string Redis parameters (like integers, booleans, or complex objects), avoid using REDIS_* environment variables as they may fail during Redis client initialization. Instead, use cache_kwargs in your router configuration for such parameters. :::

See how it's read from the environment

Step 3: Run proxy with config

$ litellm --config /path/to/config.yaml

Caching can be enabled by adding the cache key in the config.yaml

Step 1: Add `cache` to the config.yaml

model_list:
  - model_name: fake-openai-endpoint
    litellm_params:
      model: openai/fake
      api_key: fake-key
      api_base: https://exampleopenaiendpoint-production.up.railway.app/
  - model_name: openai-embedding
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  set_verbose: True
  cache: True # set cache responses to True, litellm defaults to using a redis cache
  cache_params:
    type: qdrant-semantic
    qdrant_semantic_cache_embedding_model: openai-embedding # the model should be defined on the model_list
    qdrant_collection_name: test_collection
    qdrant_quantization_config: binary
    qdrant_semantic_cache_vector_size: 1536 # vector size must match embedding model dimensionality
    similarity_threshold: 0.8 # similarity threshold for semantic cache

Step 2: Add Qdrant Credentials to your .env

QDRANT_API_KEY = "16rJUMBRx*************"
QDRANT_API_BASE = "https://5392d382-45*********.cloud.qdrant.io"

Step 3: Run proxy with config

$ litellm --config /path/to/config.yaml

Step 4. Test it

curl -i http://localhost:4000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -H "Authorization: Bearer sk-1234" \\
  -d '{
    "model": "fake-openai-endpoint",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }'

Expect to see x-litellm-semantic-similarity in the response headers when semantic caching is one

Step 1: Add `cache` to the config.yaml

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
  - model_name: text-embedding-ada-002
    litellm_params:
      model: text-embedding-ada-002

litellm_settings:
  set_verbose: True
  cache: True # set cache responses to True
  cache_params: # set cache params for s3
    type: s3
    s3_bucket_name: cache-bucket-litellm # AWS Bucket Name for S3
    s3_region_name: us-west-2 # AWS Region Name for S3
    s3_aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # us os.environ/<variable name> to pass environment variables. This is AWS Access Key ID for S3
    s3_aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY # AWS Secret Access Key for S3
    s3_endpoint_url: https://s3.amazonaws.com # [OPTIONAL] S3 endpoint URL, if you want to use Backblaze/cloudflare s3 buckets

Step 2: Run proxy with config

$ litellm --config /path/to/config.yaml

Step 1: Add `cache` to the config.yaml

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
  - model_name: text-embedding-ada-002
    litellm_params:
      model: text-embedding-ada-002

litellm_settings:
  set_verbose: True
  cache: True # set cache responses to True
  cache_params: # set cache params for gcs
    type: gcs
    gcs_bucket_name: cache-bucket-litellm # GCS Bucket Name for caching
    gcs_path_service_account: os.environ/GCS_PATH_SERVICE_ACCOUNT # use os.environ/<variable name> to pass environment variables. This is the path to your GCS service account JSON file
    gcs_path: cache/ # [OPTIONAL] GCS path prefix for cache objects

Step 2: Add GCS Credentials to .env

Set the GCS environment variables in your .env file:

GCS_BUCKET_NAME="your-gcs-bucket-name"
GCS_PATH_SERVICE_ACCOUNT="/path/to/service-account.json"

Step 3: Run proxy with config

$ litellm --config /path/to/config.yaml

Caching can be enabled by adding the cache key in the config.yaml

Step 1: Add `cache` to the config.yaml

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
  - model_name: azure-embedding-model
    litellm_params:
      model: azure/azure-embedding-model
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2023-07-01-preview"

litellm_settings:
  set_verbose: True
  cache: True # set cache responses to True
  cache_params:
    type: "redis-semantic"
    similarity_threshold: 0.8 # similarity threshold for semantic cache
    redis_semantic_cache_embedding_model: azure-embedding-model # set this to a model_name set in model_list

Step 2: Add Redis Credentials to .env

Set either REDIS_URL or the REDIS_HOST in your os environment, to enable caching.

REDIS_URL = ""        # REDIS_URL='redis://username:password@hostname:port/database'
## OR ##
REDIS_HOST = ""       # REDIS_HOST='redis-18841.c274.us-east-1-3.ec2.cloud.redislabs.com'
REDIS_PORT = ""       # REDIS_PORT='18841'
REDIS_PASSWORD = ""   # REDIS_PASSWORD='liteLlmIsAmazing'

Additional kwargs
You can pass in any additional redis.Redis arg, by storing the variable + value in your os environment, like this:

REDIS_<redis-kwarg-name> = ""

Step 3: Run proxy with config

$ litellm --config /path/to/config.yaml

Step 1: Add `cache` to the config.yaml

litellm_settings:
  cache: True
  cache_params:
    type: local

Step 2: Run proxy with config

$ litellm --config /path/to/config.yaml

Step 1: Add `cache` to the config.yaml

litellm_settings:
  cache: True
  cache_params:
    type: disk
    disk_cache_dir: /tmp/litellm-cache # OPTIONAL, default to ./.litellm_cache

Step 2: Run proxy with config

$ litellm --config /path/to/config.yaml

Usage

Basic

Send the same request twice:

curl http://0.0.0.0:4000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "write a poem about litellm!"}],
     "temperature": 0.7
   }'

curl http://0.0.0.0:4000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "write a poem about litellm!"}],
     "temperature": 0.7
   }'

Send the same request twice:

curl --location 'http://0.0.0.0:4000/embeddings' \\
  --header 'Content-Type: application/json' \\
  --data ' {
  "model": "text-embedding-ada-002",
  "input": ["write a litellm poem"]
  }'

curl --location 'http://0.0.0.0:4000/embeddings' \\
  --header 'Content-Type: application/json' \\
  --data ' {
  "model": "text-embedding-ada-002",
  "input": ["write a litellm poem"]
  }'

Dynamic Cache Controls

Parameter	Type	Description

Caching

Overview

Caching

Supported Caches

Quick Start

Step 1: Add cache to the config.yaml

[OPTIONAL] Step 1.5: Add redis namespaces, default ttl

Namespace

Redis Cluster

Redis Sentinel

TTL

SSL

GCP IAM Authentication

Step 2: Add Redis Credentials to .env

Step 3: Run proxy with config

Step 1: Add cache to the config.yaml

Step 2: Add Qdrant Credentials to your .env

Step 3: Run proxy with config

Step 4. Test it

Step 1: Add cache to the config.yaml

Step 2: Run proxy with config

Step 1: Add cache to the config.yaml

Step 2: Add GCS Credentials to .env

Step 3: Run proxy with config

Step 1: Add cache to the config.yaml

Step 2: Add Redis Credentials to .env

Step 3: Run proxy with config

Step 1: Add cache to the config.yaml

Step 2: Run proxy with config

Step 1: Add cache to the config.yaml

Step 2: Run proxy with config

Usage

Basic

Dynamic Cache Controls

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml

Step 1: Add `cache` to the config.yaml