Gemini Embedding 2 (GA): Multimodal Embeddings

Litellm now fully supports Gemini Embedding 2 GA.

:::info For end-to-end behavior, input shapes, and MIME types, see the Gemini Embedding 2 Preview walkthrough. This post focuses on GA naming, cost map coverage. :::

Supported Input Types

Input Formats

LiteLLM accepts three input formats for multimodal content:

1. Data URIs – Base64-encoded inline: data:image/png;base64,<encoded_data>
2. GCS URLs – Cloud Storage paths (Vertex AI): gs://bucket/path/to/file.png
3. Gemini File References – Pre-uploaded files (Gemini API): files/abc123

Quick Start

from litellm import embedding

os.environ["GEMINI_API_KEY"] = "your-api-key"

# Text + Image (base64)
response = embedding(
    model="gemini/gemini-embedding-2",
    input=[
        "The food was delicious and the waiter...",
        "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
    ],
)
print(response)

from litellm import embedding

litellm.vertex_project = "your-project-id"
litellm.vertex_location = "us-central1"

# Text + Image (GCS URL)
response = embedding(
    model="vertex_ai/gemini-embedding-2",
    input=[
        "Describe this image",
        "gs://my-bucket/images/photo.png"
    ],
)
print(response)

1. Config (config.yaml)

model_list:
  - model_name: gemini-embedding-2
    litellm_params:
      model: gemini/gemini-embedding-2
      api_key: os.environ/GEMINI_API_KEY
  - model_name: vertex-gemini-embedding-2
    litellm_params:
      model: vertex_ai/gemini-embedding-2
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: global

general_settings:
  master_key: sk-1234

2. Start proxy

litellm --config config.yaml

3. Call embeddings (OpenAI-compatible POST /v1/embeddings on the proxy)

curl -sS -X POST http://localhost:4000/v1/embeddings \\
  -H "Authorization: Bearer sk-1234" \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "gemini-embedding-2",
    "input": [
      "The food was delicious and the waiter...",
      "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
    ]
  }'

Input Format Examples

Supported MIME Types for Data URIs

• Images: image/png, image/jpeg
• Audio: audio/mpeg, audio/wav
• Video: video/mp4, video/quicktime
• Documents: application/pdf

GCS URL MIME Inference

For Vertex AI, MIME types are inferred from file extensions:

• .png → image/png
• .jpg / .jpeg → image/jpeg
• .mp3 → audio/mpeg
• .wav → audio/wav
• .mp4 → video/mp4
• .mov → video/quicktime
• .pdf → application/pdf

Optional Parameters

response = embedding(
    model="gemini/gemini-embedding-2",
    input=["text to embed"],
    dimensions=768,  # Optional: control output vector size
)