Helicone

Overview

<br />

We support ALL models available through Helicone's AI Gateway. Use helicone/ as a prefix when sending requests.

What is Helicone?

Helicone is an open-source observability platform for LLM applications that provides:

• Request Monitoring: Track all LLM requests with detailed metrics
• Caching: Reduce costs and latency with intelligent caching
• Rate Limiting: Control request rates per user/key
• Cost Tracking: Monitor spend across models and users
• Custom Properties: Tag requests with metadata for filtering and analysis
• Prompt Management: Version control for prompts

Required Variables

os.environ["HELICONE_API_KEY"] = ""  # your Helicone API key

Get your Helicone API key from your Helicone dashboard.

Usage - LiteLLM Python SDK

Non-streaming

from litellm import completion

os.environ["HELICONE_API_KEY"] = ""  # your Helicone API key

messages = [{"content": "What is the capital of France?", "role": "user"}]

# Helicone call - routes through Helicone gateway to OpenAI
response = completion(
    model="helicone/gpt-4",
    messages=messages
)

print(response)

Streaming

from litellm import completion

os.environ["HELICONE_API_KEY"] = ""  # your Helicone API key

messages = [{"content": "Write a short poem about AI", "role": "user"}]

# Helicone call with streaming
response = completion(
    model="helicone/gpt-4",
    messages=messages,
    stream=True
)

for chunk in response:
    print(chunk)

With Metadata (Helicone Custom Properties)

from litellm import completion

os.environ["HELICONE_API_KEY"] = ""  # your Helicone API key

response = completion(
    model="helicone/gpt-4o-mini",
    messages=[{"role": "user", "content": "What's the weather like?"}],
    metadata={
        "Helicone-Property-Environment": "production",
        "Helicone-Property-User-Id": "user_123",
        "Helicone-Property-Session-Id": "session_abc"
    }
)

print(response)

Text Completion

os.environ["HELICONE_API_KEY"] = ""  # your Helicone API key

response = litellm.completion(
    model="helicone/gpt-4o-mini",  # text completion model
    prompt="Once upon a time"
)

print(response)

Retry and Fallback Mechanisms

litellm.api_base = "https://ai-gateway.helicone.ai/"
litellm.metadata = {
    "Helicone-Retry-Enabled": "true",
    "helicone-retry-num": "3",
    "helicone-retry-factor": "2",
}

response = litellm.completion(
    model="helicone/gpt-4o-mini/openai,claude-3-5-sonnet-20241022/anthropic", # Try OpenAI first, then fallback to Anthropic, then continue with other models,
    messages=[{"role": "user", "content": "Hello"}]
)

Supported OpenAI Parameters

Helicone supports all standard OpenAI-compatible parameters:

Helicone-Specific Headers

Pass these as metadata to leverage Helicone features:

Example with headers:

response = litellm.completion(
    model="helicone/gpt-4",
    messages=[{"role": "user", "content": "Hello"}],
    metadata={
        "Helicone-Cache-Enabled": "true",
        "Helicone-Property-Environment": "production",
        "Helicone-Property-User-Id": "user_123",
        "Helicone-Session-Id": "session_abc",
        "Helicone-Prompt-Id": "prompt_v1"
    }
)

Advanced Usage

Using with Different Providers

Helicone acts as a gateway and supports multiple providers:

# Set both Helicone and Anthropic keys
os.environ["HELICONE_API_KEY"] = "your-helicone-key"

response = litellm.completion(
    model="helicone/claude-3.5-haiku/anthropic",
    messages=[{"role": "user", "content": "Hello"}]
)

Caching

Enable caching to reduce costs and latency:

response = litellm.completion(
    model="helicone/gpt-4",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    metadata={
        "Helicone-Cache-Enabled": "true"
    }
)

# Subsequent identical requests will be served from cache
response2 = litellm.completion(
    model="helicone/gpt-4",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    metadata={
        "Helicone-Cache-Enabled": "true"
    }
)

Features

Request Monitoring

• Track all requests with detailed metrics
• View request/response pairs
• Monitor latency and errors
• Filter by custom properties

Cost Tracking

• Per-model cost tracking
• Per-user cost tracking
• Cost alerts and budgets
• Historical cost analysis

Rate Limiting

• Per-user rate limits
• Per-API key rate limits
• Custom rate limit policies
• Automatic enforcement

Analytics

• Request volume trends
• Cost trends
• Latency percentiles
• Error rates

Visit Helicone Pricing for details.

Additional Resources

• Helicone Official Documentation
• Helicone Dashboard
• Helicone GitHub
• API Reference