Submit
All skills
Skillintermediate

Using Vector Stores (Knowledge Bases)

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import Image from '@theme/IdealImage';

Claude Code Knowledge Pack7/10/2026

Overview

Using Vector Stores (Knowledge Bases)

<p style={{textAlign: 'left', color: '#666'}}> Use Vector Stores with any LiteLLM supported model </p>

LiteLLM integrates with vector stores, allowing your models to access your organization's data for more accurate and contextually relevant responses.

Supported Vector Stores

Quick Start

In order to use a vector store with LiteLLM, you need to

  • Initialize litellm.vector_store_registry
  • Pass tools with vector_store_ids to the completion request. Where vector_store_ids is a list of vector store ids you initialized in litellm.vector_store_registry

LiteLLM Python SDK

LiteLLM's allows you to use vector stores in the OpenAI API spec by passing a tool with vector_store_ids you want to use


from litellm.vector_stores.vector_store_registry import VectorStoreRegistry, LiteLLM_ManagedVectorStore

# Init vector store registry
litellm.vector_store_registry = VectorStoreRegistry(
    vector_stores=[
        LiteLLM_ManagedVectorStore(
            vector_store_id="T37J8R4WTM",
            custom_llm_provider="bedrock"
        )
    ]
)

# Make a completion request with vector_store_ids parameter
response = await litellm.acompletion(
    model="anthropic/claude-3-5-sonnet", 
    messages=[{"role": "user", "content": "What is litellm?"}],
    tools=[
        {
            "type": "file_search",
            "vector_store_ids": ["T37J8R4WTM"]
        }
    ],
)

print(response.choices[0].message.content)

LiteLLM Proxy

1. Configure your vector_store_registry

In order to use a vector store with LiteLLM, you need to configure your vector_store_registry. This tells litellm which vector stores to use and api provider to use for the vector store.

model_list:
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY

vector_store_registry:
  - vector_store_name: "bedrock-litellm-website-knowledgebase"
    litellm_params:
      vector_store_id: "T37J8R4WTM"
      custom_llm_provider: "bedrock"
      vector_store_description: "Bedrock vector store for the Litellm website knowledgebase"
      vector_store_metadata:
        source: "https://www.litellm.com/docs"

On the LiteLLM UI, Navigate to Experimental > Vector Stores > Create Vector Store. On this page you can create a vector store with a name, vector store id and credentials.

2. Make a request with vector_store_ids parameter

curl http://localhost:4000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -H "Authorization: Bearer $LITELLM_API_KEY" \\
  -d '{
    "model": "claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "What is litellm?"}],
    "tools": [
        {
            "type": "file_search",
            "vector_store_ids": ["T37J8R4WTM"]
        }
    ]
  }'

from openai import OpenAI

# Initialize client with your LiteLLM proxy URL
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="your-litellm-api-key"
)

# Make a completion request with vector_store_ids parameter
response = client.chat.completions.create(
    model="claude-3-5-sonnet",
    messages=[{"role": "user", "content": "What is litellm?"}],
    tools=[
        {
            "type": "file_search",
            "vector_store_ids": ["T37J8R4WTM"]
        }
    ]
)

print(response.choices[0].message.content)

Provider Specific Guides

This section covers how to add your vector stores to LiteLLM. If you want support for a new provider, please file an issue here.

Bedrock Knowledge Bases

1. Set up your Bedrock Knowledge Base

Ensure you have a Bedrock Knowledge Base created in your AWS account with the appropriate permissions configured.

2. Add to LiteLLM UI

  1. Navigate to Tools > Vector Stores > "Add new vector store"
  2. Select "Bedrock" as the provider
  3. Enter your Bedrock Knowledge Base ID in the "Vector Store ID" field

Vertex AI RAG Engine

1. Get your Vertex AI RAG Engine ID

  1. Navigate to your RAG Engine Corpus in the Google Cloud Console
  2. Select the RAG Engine you want to integrate with LiteLLM
<div style={{margin: '20px 0', padding: '10px', border: '1px solid #ddd', borderRadius: '8px', display: 'inline-block', boxShadow: '0 2px 8px rgba(0,0,0,0.1)'}}> </div>
  1. Click the "Details" button and copy the UUID for the RAG Engine
  2. The ID should look like: 6917529027641081856
<div style={{margin: '20px 0', padding: '10px', border: '1px solid #ddd', borderRadius: '8px', display: 'inline-block', boxShadow: '0 2px 8px rgba(0,0,0,0.1)'}}> </div>

2. Add to LiteLLM UI

  1. Navigate to Tools > Vector Stores > "Add new vector store"
  2. Select "Vertex AI RAG Engine" as the provider
  3. Enter your Vertex AI RAG Engine ID in the "Vector Store ID" field
<div style={{margin: '20px 0', padding: '10px', border: '1px solid #ddd', borderRadius: '8px', display: 'inline-block', boxShadow: '0 2px 8px rgba(0,0,0,0.1)'}}> </div>

PG Vector

1. Deploy the litellm-pg-vector-store connector

LiteLLM provides a server that exposes OpenAI-compatible vector_store endpoints for PG Vector. The LiteLLM Proxy server connects to your deployed service and uses it as a vector store when querying.

  1. Follow the deployment instructions for the litellm-pg-vector-store connector here
  2. For detailed configuration options, see the configuration guide

Example .env configuration for deploying litellm-pg-vector-store:

DATABASE_URL="postgresql://neondb_owner:xxxx"
SERVER_API_KEY="sk-1234"
HOST="0.0.0.0"
PORT=8001
EMBEDDING__MODEL="text-embedding-ada-002"
EMBEDDING__BASE_URL="http://localhost:4000"
EMBEDDING__API_KEY="sk-1234"
EMBEDDING__DIMENSIONS=1536
DB_FIELDS__ID_FIELD="id"
DB_FIELDS__CONTENT_FIELD="content"
DB_FIELDS__METADATA_FIELD="metadata"
DB_FIELDS__EMBEDDING_FIELD="embedding"
DB_FIELDS__VECTOR_STORE_ID_FIELD="vector_store_id"
DB_FIELDS__CREATED_AT_FIELD="created_at"

2. Add to LiteLLM UI

Once your litellm-pg-vector-store is deployed:

  1. Navigate to Tools > Vector Stores > "Add new vector store"
  2. Select "PG Vector" as the provider
  3. Enter your API Base URL and API Key for your litellm-pg-vector-store container
    • The API Key field corresponds to the SERVER_API_KEY from your .env configuration
<div style={{margin: '20px 0', padding: '10px', border: '1px solid #ddd', borderRadius: '8px', display: 'inline-block', boxShadow: '0 2px 8px rgba(0,0,0,0.1)'}}> </div>

OpenAI Vector Stores

1. Set up your OpenAI Vector Store

  1. Create your Vector Store on the OpenAI platform
  2. Note your Vector Store ID (format: vs_687ae3b2439881918b433cb99d10662e)

2. Add to LiteLLM UI

  1. Navigate to Tools > Vector Stores > "Add new vector store"
  2. Select "OpenAI" as the provider
  3. Enter your Vector Store ID in the corresponding field
  4. Enter your OpenAI API Key in the API Key field
<div style={{margin: '20px 0', padding: '10px', border: '1px solid #ddd', borderRadius: '8px', display: 'inline-block', boxShadow: '0 2px 8px rgba(0,0,0,0.1)'}}> </div>

Advanced

Logging Vector Store Usage

LiteLLM allows you to view your vector store usage in the LiteLLM UI on the Logs page.

After completing a request with a vector store, navigate to the Logs page on LiteLLM. Here you should be able to see the query sent to the vector store and corresponding response with scores.

<p style={{textAlign: 'left', color: '#666'}}> LiteLLM Logs Page: Vector Store Usage </p>

Listing available vector stores

You can list all available vector stores using the /vector_store/list endpoint

Request:

curl -X GET "http://localhost:4000/vector_store/list" \\
  -H "Authorization: Bearer $LITELLM_API_KEY"

Response:

The response will be a list of all vector stores that are available to use with LiteLLM.

{
  "object": "list",
  "data": [
    {
      "vector_store_id": "T37J8R4WTM",
      "custom_llm_provider": "bedrock",
      "vector_store_name": "bedrock-litellm-website-knowledgebase",
      "vector_store_description": "Bedrock vector store for the Litellm website knowledgebase",
      "vector_store_metadata": {
        "source": "https://www.litellm.com/docs"
      },
      "created_at": "2023-05-03T18:21:36.462Z",
      "updated_at": "2023-05-03T18:21:36.462Z",
      "litellm_credential_name": "bedrock_credentials"
    }
  ],
  "total_count": 1,
  "current_page": 1,
  "total_pages": 1
}

Always on for a model

Use this if you want vector stores to be used by default for a specific model.

In this config, we add vector_store_ids to the claude-3-5-sonnet-with-vector-store model. This means that any request to the claude-3-5-sonnet-with-vector-store model will always use the vector store with the id T37J8R4WTM defined in the vector_store_registry.

model_list:
  - model_name: claude-3-5-sonnet-with-vector-store
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      vector_store_ids: ["T37J8R4WTM"]

vector_store_registry:
  - vector_store_name: "bedrock-litellm-website-knowledgebase"
    litellm_params:
      vector_store_id: "T37J8R4WTM"
      custom_llm_provider: "bedrock"
      vector_store_description: "Bedrock vector store for the Litellm website knowledgebase"
      vector_store_metadata:
        source: "https://www.litellm.com/docs"

How It Works

If your request includes a vector_store_ids parameter where any of the vector store ids are found in the vector_store_registry, LiteLLM will automatically use the vector store for the request.

  1. You make a completion request with the vector_store_ids parameter and any of the vector store ids are found in the litellm.vector_store_registry
  2. LiteLLM automatically:
    • Uses your last message as the query to retrieve relevant information from the Knowledge Base
    • Adds the retrieved context to your conversation
    • Sends the augmented messages to the model

Example Transformation

When you pass vector_store_ids=["YOUR_KNOWLEDGE_BASE_ID"], your request flows through these steps:

1. Original Request to LiteLLM:

{
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [
        {"role": "user", "content": "What is litellm?"}
    ],
    "vector_store_ids": ["YOUR_KNOWLEDGE_BASE_ID"]
}

2. Request to AWS Bedrock Knowledge Base:

{
    "retrievalQuery": {
        "text": "What is litellm?"
    }
}

This is sent to: https://bedrock-agent-runtime.{aws_region}.amazonaws.com/knowledgebases/YOUR_KNOWLEDGE_BASE_ID/retrieve

3. Final Request to LiteLLM:

{
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [
        {"role": "user", "content": "What is litellm?"},
        {"role": "user", "content": "Context: \
\
LiteLLM is an open-source SDK to simplify LLM API calls across providers (OpenAI, Claude, etc). It provides a standardized interface with robust error handling, streaming, and observability tools."}
    ]
}

This process happens automatically whenever you include the vector_store_ids parameter in your request.

Accessing Search Results (Citations)

When using vector stores, LiteLLM automatically returns search results in provider_specific_fields. This allows you to show users citations for the AI's response.

Key Concept

Search results are always in: response.choices[0].message.provider_specific_fields["search_results"]

For streaming: Results appear in the final chunk when finish_reason == "stop"

Non-Streaming Example

Non-Streaming Response with search results:

{
  "id": "chatcmpl-abc123",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "LiteLLM is a platform...",
      "provider_specific_fields": {
        "search_results": [{
          "search_query": "What is litellm?",
          "data": [{
            "score": 0.95,
            "content": [{"text": "...", "type": "text"}],
            "filename": "litellm-docs.md",
            "file_id": "doc-123"
          }]
        }]
      }
    },
    "finish_reason": "stop"
  }]
}

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api