Skillintermediate

Docker, Helm, Terraform

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import Image from '@theme/IdealImage';

Claude Code Knowledge Pack7/10/2026

Overview

Docker, Helm, Terraform

:::info No Limits on LiteLLM OSS There are no limits on the number of users, keys, or teams you can create on LiteLLM OSS. :::

You can find the Dockerfile to build litellm proxy here

Note: Production requires at least 4 CPU cores and 8 GB RAM.

Quick Start

:::info Facing issues with pulling the docker image? Email us at support@berri.ai. :::

To start using Litellm, run the following commands in a shell:

docker pull docker.litellm.ai/berriai/litellm:main-latest

See all docker images

$ uv tool install 'litellm[proxy]'

Use this docker compose to spin up the proxy with a postgres database running locally.

# Get the docker compose file
curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/prometheus.yml

# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env

# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/ 
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env

# Start
docker compose up

Verify Docker image signatures

All LiteLLM Docker images are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \\
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \\
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \\
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \\
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Expected output:

The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Learn more about LiteLLM's release signing in the CI/CD v2 announcement. For a complete guide covering all image variants, CI/CD enforcement, and deployment best practices, see the Docker Image Security Guide.

Docker Run

Step 1. CREATE config.yaml

Example litellm_config.yaml

model_list:
  - model_name: azure-gpt-4o
    litellm_params:
      model: azure/<your-azure-model-deployment>
      api_base: os.environ/AZURE_API_BASE # runs os.getenv("AZURE_API_BASE")
      api_key: os.environ/AZURE_API_KEY # runs os.getenv("AZURE_API_KEY")
      api_version: "2025-01-01-preview"

Step 2. RUN Docker Image

docker run \\
    -v $(pwd)/litellm_config.yaml:/app/config.yaml \\
    -e AZURE_API_KEY=d6*********** \\
    -e AZURE_API_BASE=https://openai-***********/ \\
    -p 4000:4000 \\
    docker.litellm.ai/berriai/litellm:main-stable \\
    --config /app/config.yaml --detailed_debug

Get Latest Image 👉 here

Step 3. TEST Request

Pass model=azure-gpt-4o this was set on step 1

curl --location 'http://0.0.0.0:4000/chat/completions' \\
    --header 'Content-Type: application/json' \\
    --data '{
    "model": "azure-gpt-4o",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
}'

Docker Run - CLI Args

See all supported CLI args here:

Here's how you can run the docker image and pass your config to litellm

docker run docker.litellm.ai/berriai/litellm:main-stable --config your_config.yaml

Here's how you can run the docker image and start litellm on port 8002 with num_workers=8

docker run docker.litellm.ai/berriai/litellm:main-stable --port 8002 --num_workers 8

Use litellm as a base image

# Use the provided base image
FROM docker.litellm.ai/berriai/litellm:main-stable

# Set the working directory to /app
WORKDIR /app

# Copy the configuration file into the container at /app
COPY config.yaml .

# Make sure your docker/entrypoint.sh is executable
RUN chmod +x ./docker/entrypoint.sh

# Expose the necessary port
EXPOSE 4000/tcp

# Override the CMD instruction with your desired command and arguments
# WARNING: FOR PROD DO NOT USE `--detailed_debug` it slows down response times, instead use the following CMD
# CMD ["--port", "4000", "--config", "config.yaml"]

CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"]

Build from published LiteLLM packages

Follow these instructions to build a Docker container from published LiteLLM packages. If your company has a strict requirement around security or image provenance, you can follow these steps.

Note: Copy the schema.prisma file from the LiteLLM repository into your build directory alongside this Dockerfile.

Dockerfile

FROM cgr.dev/chainguard/python:latest-dev
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.10.9

USER root
WORKDIR /app

ENV UV_TOOL_BIN_DIR=/usr/local/bin

# Install runtime dependencies
RUN apk update && \\
    apk add --no-cache gcc python3-dev openssl openssl-dev

COPY --from=$UV_IMAGE /uv /usr/local/bin/uv
COPY --from=$UV_IMAGE /uvx /usr/local/bin/uvx

RUN uv tool install 'litellm[proxy,proxy-runtime,extra_proxy]==1.57.3' \\
    --python python

# Copy Prisma schema file
COPY schema.prisma .

# Generate prisma client
RUN prisma generate

EXPOSE 4000/tcp

ENTRYPOINT ["litellm"]
CMD ["--port", "4000"]

Build the docker image

docker build \\
  -f Dockerfile \\
  -t litellm-proxy-from-package-5 .

Run the docker image

docker run \\
    -v $(pwd)/litellm_config.yaml:/app/config.yaml \\
    -e OPENAI_API_KEY="sk-1222" \\
    -e DATABASE_URL="postgresql://xxxxxxxxx \\
    -p 4000:4000 \\
    litellm-proxy-from-package-5 \\
    --config /app/config.yaml --detailed_debug

Terraform

s/o Nicholas Cecere for his LiteLLM User Management Terraform

👉 Go here for Terraform

Kubernetes

Deploying a config file based litellm instance just requires a simple deployment that loads the config.yaml file via a config map. Also it would be a good practice to use the env var declaration for api keys, and attach the env vars with the api key values as an opaque secret.

apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config-file
data:
  config.yaml: |
      model_list: 
        - model_name: gpt-4o
          litellm_params:
            model: azure/gpt-4o-ca
            api_base: https://my-endpoint-canada-berri992.openai.azure.com/
            api_key: os.environ/CA_AZURE_OPENAI_API_KEY
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: litellm-secrets
data:
  CA_AZURE_OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
  labels:
    app: litellm
spec:
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: docker.litellm.ai/berriai/litellm:main-stable # it is recommended to fix a version generally
        args:
          - "--config"
          - "/app/proxy_server_config.yaml"
        ports:
        - containerPort: 4000
        volumeMounts:
        - name: config-volume
          mountPath: /app/proxy_server_config.yaml
          subPath: config.yaml
        envFrom:
        - secretRef:
            name: litellm-secrets
      volumes:
        - name: config-volume
          configMap:
            name: litellm-config-file

:::info To avoid issues with predictability, difficulties in rollback, and inconsistent environments, use versioning or SHA digests (for example, litellm:main-v1.30.3 or litellm@sha256:12345abcdef...) instead of litellm:main-stable. :::

Helm Chart

:::info

[BETA] Helm Chart is BETA. If you run into an issues/have feedback please let us know https://github.com/BerriAI/litellm/issues

:::

Use this when you want to use litellm helm chart as a dependency for other charts. The litellm-helm OCI is hosted here https://github.com/BerriAI/litellm/pkgs/container/litellm-helm

Step 1. Pull the litellm helm chart

helm pull oci://docker.litellm.ai/berriai/litellm-helm

# Pulled: docker.litellm.ai/berriai/litellm-helm:0.1.2
# Digest: sha256:7d3ded1c99c1597f9ad4dc49d84327cf1db6e0faa0eeea0c614be5526ae94e2a

Step 2. Unzip litellm helm

Unzip the specific version that was pulled in Step 1

tar -zxvf litellm-helm-0.1.2.tgz

Step 3. Install litellm helm

helm install lite-helm ./litellm-helm

Step 4. Expose the service to localhost

kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

Your LiteLLM Proxy Server is now running on http://127.0.0.1:4000.

That's it ! That's the quick start to deploy litellm

Make LLM API Requests

:::info 💡 Go here 👉 to make your first LLM API Request

LiteLLM is compatible with several SDKs - including OpenAI SDK, Anthropic SDK, Mistral SDK, LLamaIndex, Langchain (Js, Python)

:::

Deployment Options

Docs	When to Use
Quick Start	call 100+ LLMs + Load Balancing
Deploy with Database	+ use Virtual Keys + Track Spend (Note: When deploying with a database providing a `DATABASE_URL` and `LITELLM_MASTER_KEY` are required in your env )
LiteLLM container + Redis	+ load balance across multiple litellm containers
LiteLLM Database container + PostgresDB + Redis	+ use Virtual Keys + Track Spend + load balance across multiple litellm containers

Deploy with Database

Docker, Kubernetes, Helm Chart

:::warning High Traffic Deployments (1000+ RPS)

If you expect high traffic (1000+ requests per second), Redis is required to prevent database connection exhaustion and deadlocks.

Add this to your config:

general_settings:
  use_redis_transaction_buffer: true

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: your-redis-host

See Resolve DB Deadlocks for details.

:::

Requirements:

Need a postgres database (e.g. Supabase, Neon, etc) Set DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> in your env
Set a LITELLM_MASTER_KEY, this is your Proxy Admin key - you can use this to create other keys (🚨 must start with sk-)

We maintain a separate Dockerfile for reducing build time when running LiteLLM proxy with a connected Postgres Database

docker pull docker.litellm.ai/berriai/litellm-database:main-stable

docker run \\
    -v $(pwd)/litellm_config.yaml:/app/config.yaml \\
    -e LITELLM_MASTER_KEY=sk-1234 \\
    -e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \\
    -e AZURE_API_KEY=d6*********** \\
    -e AZURE_API_BASE=https://openai-***********/ \\
    -p 4000:4000 \\
    docker.litellm.ai/berriai/litellm-database:main-stable \\
    --config /app/config.yaml --detailed_debug

Your LiteLLM Proxy Server is now running on http://0.0.0.0:4000.

Step 1. Create deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
        - name: litellm-container
          image: docker.litellm.ai/berriai/litellm:main-stable
          imagePullPolicy: Always
          env:
            - name: AZURE_API_KEY
              value: "d6******"
            - name: AZURE_API_BASE
              value: "https://ope******"
            - name: LITELLM_MASTER_KEY
              value: "sk-1234"
            - name: DATABASE_URL
              value: "po**********"
          args:
            - "--config"
            - "/app/proxy_config.yaml"  # Update the path to mount the config file
          volumeMounts:                 # Define volume mount for proxy_config.yaml
            - name: config-volume
              mountPath: /app/proxy_config.yaml
              subPath: config.yaml      # Specify the field under data of the ConfigMap litellm-config
              readOnly: true
          livenessProbe: