📈 Prometheus metrics

LiteLLM Exposes a /metrics endpoint for Prometheus to Poll

Quick Start

If you're using the LiteLLM CLI with litellm --config proxy_config.yaml then you need to uv add prometheus_client==0.20.0. This is already pre-installed on the litellm Docker image

Add this to your proxy config.yaml

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
litellm_settings:
  callbacks:
    - prometheus

Start the proxy

litellm --config config.yaml --debug

Test Request

curl --location 'http://0.0.0.0:4000/chat/completions' \\
    --header 'Content-Type: application/json' \\
    --data '{
    "model": "gpt-4o",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ]
}'

View Metrics on /metrics, Visit http://localhost:4000/metrics

http://localhost:4000/metrics

# <proxy_base_url>/metrics

Multiple Workers

When using LiteLLM with multiple workers, you need to set the PROMETHEUS_MULTIPROC_DIR environment variable to enable aggregated metric collection across worker processes.

This directory is used by the Prometheus client library to store metric files that can be shared across multiple worker processes. Make sure the directory exists and is writable by your LiteLLM process.

Virtual Keys, Teams, Internal Users

Use this for for tracking per user, key, team, etc.

Team - Budget

Virtual Key - Budget

Virtual Key - Rate Limit

Initialize Budget Metrics on Startup

If you want litellm to emit the budget metrics for all keys, teams irrespective of whether they are getting requests or not, set prometheus_initialize_budget_metrics to true in the config.yaml

How this works:

• If the prometheus_initialize_budget_metrics is set to true
  • Every 5 minutes litellm runs a cron job to read all keys, teams from the database
  • It then emits the budget metrics for each key, team
  • This is used to populate the budget metrics on the /metrics endpoint

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_initialize_budget_metrics: true

Pod Health Metrics

Use these to measure per-pod queue depth and diagnose latency that occurs before LiteLLM starts processing a request.

When to use this

LiteLLM measures latency from when its handler starts. If a request waits in uvicorn's event loop before the handler runs, that wait is invisible to LiteLLM's own logs. litellm_in_flight_requests shows how loaded the pod was at any point in time.

high in_flight_requests + high ALB TargetResponseTime → pod overloaded, scale out
low  in_flight_requests + high ALB TargetResponseTime → delay is pre-ASGI (event loop blocking)

You can also check the current value directly without Prometheus:

curl http://localhost:4000/health/backlog \\
  -H "Authorization: Bearer sk-..."
# {"in_flight_requests": 47}

Proxy Level Tracking Metrics

Use this to track overall LiteLLM Proxy usage.

• Track Actual traffic rate to proxy
• Number of client side requests and failures for requests made to proxy

Callback Logging Metrics

Monitor failures while shipping logs to downstream callbacks like s3_v3 cold storage

Supported Callbacks:

• S3Logger - S3 v2 cold storage failures
• langfuse - Langfuse logging failures
• otel - OpenTelemetry logging failures

LLM Provider Metrics

Use this for LLM API Error monitoring and tracking remaining rate limits and token limits

Labels Tracked

Success and Failure

Remaining Requests and Tokens

Deployment State

Fallback (Failover) Metrics

Request Counting Metrics

Request Latency Metrics

Tracking end_user on Prometheus

By default LiteLLM does not track end_user on Prometheus. This is done to reduce the cardinality of the metrics from LiteLLM Proxy.

If you want to track end_user on Prometheus, you can do the following:

litellm_settings:
  callbacks: ["prometheus"]
  enable_end_user_cost_tracking_prometheus_only: true

Emit Stream Label

Add a stream label to litellm_proxy_total_requests_metric to split requests by streaming vs. non-streaming. Disabled by default.

litellm_settings:
  callbacks: ["prometheus"]
  prometheus_emit_stream_label: true

When enabled, litellm_proxy_total_requests_metric gains a stream label with values "True", "False", or "None".

litellm_proxy_total_requests_metric{..., stream="True"} 42
litellm_proxy_total_requests_metric{..., stream="False"} 100

:::note This label is opt-in because adding a new label to an existing metric changes its cardinality and breaks existing Prometheus queries / Grafana dashboards that target this metric. Enable it only on fresh deployments or when you are ready to update your dashboards. :::

[BETA] Custom Metrics

Track custom metrics on prometheus on all events mentioned above.

Custom Metadata Labels

1. Define the custom metadata labels in the config.yaml

model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  custom_prometheus_metadata_labels: ["metadata.foo", "metadata.bar"]

2. Make a request with the custom metadata labels

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completi