SAP Generative AI Hub

LiteLLM supports SAP Generative AI Hub's Orchestration Service.

Prerequisites

Before you begin, ensure you have:

1. SAP BTP Account with access to SAP AI Core
2. AI Core Service Instance provisioned in your subaccount
3. Service Key created for your AI Core instance (this contains your credentials)
4. Resource Group with deployed AI models (check with your SAP administrator)

:::tip Where to Find Your Credentials Your credentials come from the Service Key you create in SAP BTP Cockpit:

1. Navigate to your Subaccount → Instances and Subscriptions
2. Find your AI Core instance and click on it
3. Go to Service Keys and create one (or use existing)
4. The JSON contains all values needed below

The service key JSON looks like this:

{
  "clientid": "sb-abc123...",
  "clientsecret": "xyz789...",
  "url": "https://myinstance.authentication.eu10.hana.ondemand.com",
  "serviceurls": {
    "AI_API_URL": "https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com"
  }
}

:::info Resource Group The resource group is typically configured separately in your AI Core deployment, not in the service key itself. You can set it via the AICORE_RESOURCE_GROUP environment variable (defaults to "default"). :::

Quick Start

Step 1: Install LiteLLM

uv add litellm

Step 2: Set Your Credentials

Choose one of these authentication methods:

Breaking change: credential resolution is "first-source-wins"

Credential resolution no longer merges individual fields across sources.

Resolution order is: kwargs → service key → env (AICORE_*) → config → VCAP service

Important behavior: once LiteLLM finds any credential value in a source, it takes all credentials from that source exclusively (except resource_group, which may still be resolved separately).

The simplest approach - paste your entire service key as a single environment variable.

Note: the service key no more needs to be wrapped in a "credentials" key.

    "clientid": "your-client-id",
    "clientsecret": "your-client-secret",
    "url": "https://<your-instance>.authentication.sap.hana.ondemand.com",
    "serviceurls": {
      "AI_API_URL": "https://api.ai.<your-region>.aws.ml.hana.ondemand.com"
    }
}'

Alternatively, instead of using the service key above, you could set each credential separately:

Step 3: Make Your First Request

from litellm import completion

response = completion(
    model="sap/gpt-4o",
    messages=[{"role": "user", "content": "Hello from LiteLLM!"}]
)
print(response.choices[0].message.content)

Run it:

python test_sap.py

Expected output:

Hello! How can I assist you today?

Step 4: Verify Your Setup (Optional)

Test that everything is working with this diagnostic script:

# Enable debug logging to see what's happening

os.environ["LITELLM_LOG"] = "DEBUG"

# Either use AICORE_SERVICE_KEY (contains all credentials including resourcegroup)
# OR use individual variables (all required together)
individual_vars = ["AICORE_AUTH_URL", "AICORE_CLIENT_ID", "AICORE_CLIENT_SECRET", "AICORE_BASE_URL", "AICORE_RESOURCE_GROUP"]

print("=== SAP Gen AI Hub Setup Verification ===\
")

# Check for service key method
if os.environ.get("AICORE_SERVICE_KEY"):
    print("✓ Using AICORE_SERVICE_KEY authentication (includes resource group)")
else:
    # Check individual variables
    missing = [v for v in individual_vars if not os.environ.get(v)]
    if missing:
        print(f"✗ Missing environment variables: {missing}")
    else:
        print("✓ Using individual variable authentication")
        print(f"✓ Resource group: {os.environ.get('AICORE_RESOURCE_GROUP')}")

# Test API connection
print("\
=== Testing API Connection ===\
")
try:
    response = litellm.completion(
        model="sap/gpt-4o",
        messages=[{"role": "user", "content": "Say 'Connection successful!' and nothing else."}],
        max_tokens=20
    )
    print(f"✓ API Response: {response.choices[0].message.content}")
    print("\
🎉 Setup complete! You're ready to use SAP Gen AI Hub with LiteLLM.")
except Exception as e:
    print(f"✗ API Error: {e}")
    print("\
Troubleshooting tips:")
    print("  1. Verify your service key credentials are correct")
    print("  2. Check that 'gpt-4o' is deployed in your resource group")
    print("  3. Ensure your SAP AI Core instance is running")

Run the verification:

python verify_sap_setup.py

Expected output on success:

=== SAP Gen AI Hub Setup Verification ===

✓ Using AICORE_SERVICE_KEY authentication
✓ Resource group: default

=== Testing API Connection ===

✓ API Response: Connection successful!

🎉 Setup complete! You're ready to use SAP Gen AI Hub with LiteLLM.

Authentication

SAP Generative AI Hub uses OAuth2 service keys for authentication. See Quick Start for setup instructions.

Environment Variables Reference

*Choose either AICORE_SERVICE_KEY OR the individual variables (AICORE_AUTH_URL, AICORE_CLIENT_ID, AICORE_CLIENT_SECRET, AICORE_BASE_URL).

Model Naming Conventions

Understanding model naming is crucial for using SAP Gen AI Hub correctly. The naming pattern differs depending on whether you're using the SDK directly or through the proxy.

Direct SDK Usage

When calling LiteLLM's SDK directly, you must include the sap/ prefix in the model name:

# Correct - includes sap/ prefix
model="sap/gpt-4o"
model="sap/anthropic--claude-4.5-sonnet"
model="sap/gemini-2.5-pro"

# Incorrect - missing prefix
model="gpt-4o"  # ❌ Won't work

3. Environment variables - Set the following list of credentials in .env file

<pre> AICORE_AUTH_URL = "https://* * * .authentication.sap.hana.ondemand.com/oauth/token", AICORE_CLIENT_ID = " *** ", AICORE_CLIENT_SECRET = " *** ", AICORE_RESOURCE_GROUP = " *** ", AICORE_BASE_URL = "https://api.ai.***.cfapps.sap.hana.ondemand.com/v2" </pre>

Other credential configuration options are also available. For more information, see the SAP AI Core Documentation.

Usage - LiteLLM Python SDK

Proxy Usage

When using the LiteLLM Proxy, you use the friendly model_name defined in your configuration. The proxy automatically handles the sap/ prefix routing.

# In config.yaml, define the mapping
model_list:
  - model_name: gpt-4o          # ← Use this name in client requests
    litellm_params:
      model: sap/gpt-4o         # ← Proxy handles the sap/ prefix

# Client request - no sap/ prefix needed
client.chat.completions.create(
    model="gpt-4o",  # ✓ Correct for proxy usage
    messages=[...]
)

Anthropic Models Special Syntax

Anthropic models use a double-dash (--) prefix convention:

Quick Reference Table

Using the Python SDK

The LiteLLM Python SDK automatically detects your authentication method. Simply set your environment variables and make requests.

from litellm import completion

# Assumes AICORE_AUTH_URL, AICORE_CLIENT_ID, etc. are set
response = completion(
    model="sap/anthropic--claude-4.5-sonnet",
    messages=[{"role": "user", "content": "Explain quantum computing"}]
)
print(response.choices[0].message.content)

Both authentication methods (individual variables or service key JSON) work automatically - no code changes required.

Using the Proxy Server

The LiteLLM Proxy provides a unified OpenAI-compatible API for your SAP models.

Configuration

Create a config.yaml file in your project directory with your model mappings and credentials:

model_list:
  # OpenAI models
  - model_name: gpt-5
    litellm_params:
      model: sap/gpt-5

  # Anthropic models (note the double-dash)
  - model_name: claude-sonnet
    litellm_params:
      model: sap/anthropic--claude-4.5-sonnet

  - model_name: claude-opus
    litellm_params:
      model: sap/anthropic--claude-4.5-opus

  # Embeddings
  - model_name: text-embedding-3-small
    litellm_params:
      model: sap/text-embedding-3-small

litellm_settings:
  drop_params: true
  set_verbose: false
  request_timeout: 600
  num_retries: 2
  forward_client_headers_to_llm_api: ["anthropic-version"]

general_settings:
  master_key: "sk-1234" # Enter here your desired master key starting with 'sk-'.
  
  # UI Admin is not required but helpful including the management of keys for your team(s). If you are using a database, these parameters are required:
  database_url: "Enter you database URL."
  UI_USERNAME: "Your desired UI admin account name"
  UI_PASSWORD: "Your desired and strong pwd"

# Authentication
environment_variables:
  AICORE_SERVICE_KEY: '{"credentials": {"clientid": "...", "clientsecret": "...", "url": "...", "serviceurls": {"AI_API_URL": "..."}}}'
  AICORE_RESOURCE_GROUP: "default"

Starting the Proxy

litellm --config config.yaml

The proxy will start on http://localhost:4000 by default.

Making Requests

curl http://localhost:4000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -H "Authorization: Bearer sk-1234" \\
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-1234"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

os.environ["LITELLM_PROXY_API_KEY"] = "sk-1234"
litellm.use_litellm_proxy = True

response = litellm.completion(
    model="claude-sonnet",
    messages=[{"content": "Hello, how are you?", "role": "user"}],
    api_base="http://localhost:4000"
)

print(response)

Features

Streaming Responses

Stream responses in real-time for better user experience:

from litellm import completion

response = completion(
    model="sap/gpt-4o",
    messages=[{"role": "user", "content": "Count from 1 to 10"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Structured Output

JSON Schema (Recommended)

Use JSON Schema for structured output with strict validation:

from litellm import completion

response = completion(
    model="sap/gpt-4o",
    messages=[{
        "role": "user",
        "content": "Generate info about Tokyo"
    }],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "city_info",
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "population": {"type": "number"},
                    "country": {"type": "string"}
                },
                "required": ["name", "population", "country"],
                "additionalProperties": False
            },
            "strict": True
        }
    }
)

print(response.choices[0].message.content)
# Output: {"name":"Tokyo","population":37000000,"country":"Japan"}

JSON Object Format

For flexible JSON output without schema validation:

from litellm import completion

response = completion(
    model="sap/gpt-4o",
    messages=[{
        "role": "user",
        "content": "Generat