Chutes

Overview

<br />

What is Chutes?

Chutes is a powerful AI deployment and serving platform that provides:

• Pre-built Templates: Ready-to-use configurations for vLLM, SGLang, diffusion models, and embeddings
• OpenAI-Compatible APIs: Use standard OpenAI SDKs and clients
• Multi-GPU Scaling: Support for large models across multiple GPUs
• Streaming Responses: Real-time model outputs
• Custom Configurations: Override any parameter for your specific needs
• Performance Optimization: Pre-configured optimization settings

Required Variables

os.environ["CHUTES_API_KEY"] = ""  # your Chutes API key

Get your Chutes API key from chutes.ai.

Usage - LiteLLM Python SDK

Non-streaming

from litellm import completion

os.environ["CHUTES_API_KEY"] = ""  # your Chutes API key

messages = [{"content": "What is the capital of France?", "role": "user"}]

# Chutes call
response = completion(
    model="chutes/model-name",  # Replace with actual model name
    messages=messages
)

print(response)

Streaming

from litellm import completion

os.environ["CHUTES_API_KEY"] = ""  # your Chutes API key

messages = [{"content": "Write a short poem about AI", "role": "user"}]

# Chutes call with streaming
response = completion(
    model="chutes/model-name",  # Replace with actual model name
    messages=messages,
    stream=True
)

for chunk in response:
    print(chunk)

Usage - LiteLLM Proxy Server

1. Save key in your environment

2. Start the proxy

model_list:
  - model_name: chutes-model
    litellm_params:
      model: chutes/model-name  # Replace with actual model name
      api_key: os.environ/CHUTES_API_KEY

Supported OpenAI Parameters

Chutes supports all standard OpenAI-compatible parameters:

Support Frameworks

Chutes provides optimized templates for popular AI frameworks:

vLLM (High-Performance LLM Serving)

• OpenAI-compatible endpoints
• Multi-GPU scaling support
• Advanced optimization settings
• Best for production workloads

SGLang (Advanced LLM Serving)

• Structured generation capabilities
• Advanced features and controls
• Custom configuration options
• Best for complex use cases

Diffusion Models (Image Generation)

• Pre-configured image generation templates
• Optimized settings for best results
• Support for popular diffusion models

Embedding Models

• Text embedding templates
• Vector search optimization
• Support for popular embedding models

Authentication

Chutes supports multiple authentication methods:

• API Key via X-API-Key header
• Bearer token via Authorization header

Example for LiteLLM (uses environment variable):

os.environ["CHUTES_API_KEY"] = "your-api-key"

Performance Optimization

Chutes offers hardware selection and optimization:

• Small Models (7B-13B): 1 GPU with 24GB VRAM
• Medium Models (30B-70B): 4 GPUs with 80GB VRAM each
• Large Models (100B+): 8 GPUs with 140GB+ VRAM each

Engine optimization parameters available for fine-tuning performance.

Deployment Options

Chutes provides flexible deployment:

• Quick Setup: Use pre-built templates for instant deployment
• Custom Images: Deploy with custom Docker images
• Scaling: Configure max instances and auto-scaling thresholds
• Hardware: Choose specific GPU types and configurations

Additional Resources

• Chutes Documentation
• Chutes Getting Started
• Chutes API Reference