Azure Document Intelligence OCR

Overview

Extract text and analyze document structure using Azure Document Intelligence's powerful prebuilt models.

Quick Start

LiteLLM SDK

# Set environment variables
os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

# OCR with PDF URL
response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

# Access extracted text
for page in response.pages:
    print(f"Page {page.index}:")
    print(page.markdown)

LiteLLM PROXY

model_list:
  - model_name: azure-doc-intel
    litellm_params:
      model: azure_ai/doc-intelligence/prebuilt-layout
      api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
      api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
    model_info:
      mode: ocr

Start Proxy

litellm --config proxy_config.yaml

Call OCR via Proxy

curl -X POST http://localhost:4000/ocr \\
  -H "Content-Type: application/json" \\
  -H "Authorization: Bearer your-api-key" \\
  -d '{
    "model": "azure-doc-intel",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    }
  }'

How It Works

Azure Document Intelligence uses an asynchronous API pattern. LiteLLM AI Gateway handles the request/response transformation and polling automatically.

Complete Flow Diagram

sequenceDiagram
    participant Client
    box rgb(200, 220, 255) LiteLLM AI Gateway
        participant LiteLLM
    end
    participant Azure as Azure Document Intelligence

    Client->>LiteLLM: POST /ocr (Mistral format)
    Note over LiteLLM: Transform to Azure format
    
    LiteLLM->>Azure: POST :analyze
    Azure-->>LiteLLM: 202 Accepted + polling URL
    
    Note over LiteLLM: Automatic Polling
    loop Every 2-10 seconds
        LiteLLM->>Azure: GET polling URL
        Azure-->>LiteLLM: Status: running
    end
    
    LiteLLM->>Azure: GET polling URL
    Azure-->>LiteLLM: Status: succeeded + results
    
    Note over LiteLLM: Transform to Mistral format
    LiteLLM-->>Client: OCR Response (Mistral format)

What LiteLLM Does For You

When you call litellm.ocr() via SDK or /ocr via Proxy:

1. Request Transformation: Converts Mistral OCR format → Azure Document Intelligence format
2. Submits Document: Sends transformed request to Azure DI API
3. Handles 202 Response: Captures the Operation-Location URL from response headers
4. Automatic Polling:
   • Polls the operation URL at intervals specified by retry-after header (default: 2 seconds)
   • Continues until status is succeeded or failed
   • Respects Azure's rate limiting via retry-after headers
5. Response Transformation: Converts Azure DI format → Mistral OCR format
6. Returns Result: Sends unified Mistral format response to client

Polling Configuration:

• Default timeout: 120 seconds
• Configurable via AZURE_OPERATION_POLLING_TIMEOUT environment variable
• Uses sync (time.sleep()) or async (await asyncio.sleep()) based on call type

:::info Typical processing time: 2-10 seconds depending on document size and complexity :::

Supported Models

Azure Document Intelligence offers several prebuilt models optimized for different use cases:

prebuilt-layout (Recommended)

Best for general document OCR with structure preservation.

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

model_list:
  - model_name: azure-layout
    litellm_params:
      model: azure_ai/doc-intelligence/prebuilt-layout
      api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
      api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
    model_info:
      mode: ocr

Usage:

curl -X POST http://localhost:4000/ocr \\
  -H "Authorization: Bearer your-api-key" \\
  -d '{"model": "azure-layout", "document": {"type": "document_url", "document_url": "https://example.com/doc.pdf"}}'

Features:

• Text extraction with markdown formatting
• Table detection and extraction
• Document structure analysis
• Paragraph and section recognition

Pricing: $10 per 1,000 pages

prebuilt-read

Optimized for reading text from documents - fastest and most cost-effective.

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-read",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

model_list:
  - model_name: azure-read
    litellm_params:
      model: azure_ai/doc-intelligence/prebuilt-read
      api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
      api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
    model_info:
      mode: ocr

Usage:

curl -X POST http://localhost:4000/ocr \\
  -H "Authorization: Bearer your-api-key" \\
  -d '{"model": "azure-read", "document": {"type": "document_url", "document_url": "https://example.com/doc.pdf"}}'

Features:

• Fast text extraction
• Optimized for reading-heavy documents
• Basic structure recognition

Pricing: $1.50 per 1,000 pages

prebuilt-document

General-purpose document analysis with key-value pairs.

os.environ["AZURE_DOCUMENT_INTELLIGENCE_API_KEY"] = "your-api-key"
os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] = "https://your-resource.cognitiveservices.azure.com"

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-document",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

model_list:
  - model_name: azure-document
    litellm_params:
      model: azure_ai/doc-intelligence/prebuilt-document
      api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
      api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
    model_info:
      mode: ocr

Usage:

curl -X POST http://localhost:4000/ocr \\
  -H "Authorization: Bearer your-api-key" \\
  -d '{"model": "azure-document", "document": {"type": "document_url", "document_url": "https://example.com/doc.pdf"}}'

Pricing: $10 per 1,000 pages

Document Types

Azure Document Intelligence supports various document formats.

PDF Documents

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": "https://example.com/document.pdf"
    }
)

Image Documents

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "image_url",
        "image_url": "https://example.com/image.png"
    }
)

Supported image formats: JPEG, PNG, BMP, TIFF

Base64 Encoded Documents

# Read and encode PDF
with open("document.pdf", "rb") as f:
    pdf_base64 = base64.b64encode(f.read()).decode()

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={
        "type": "document_url",
        "document_url": f"data:application/pdf;base64,{pdf_base64}"
    }
)

Response Format

# Response has the following structure
response.pages          # List of pages with extracted text
response.model          # Model used
response.object         # "ocr"
response.usage_info     # Token usage information

# Access page content
for page in response.pages:
    print(f"Page {page.index}:")
    print(page.markdown)
    
    # Page dimensions (in pixels)
    if page.dimensions:
        print(f"Width: {page.dimensions.width}px")
        print(f"Height: {page.dimensions.height}px")

Async Support

async def process_document():
    response = await litellm.aocr(
        model="azure_ai/doc-intelligence/prebuilt-layout",
        document={
            "type": "document_url",
            "document_url": "https://example.com/document.pdf"
        }
    )
    return response

# Run async function
response = asyncio.run(process_document())

Cost Tracking

LiteLLM automatically tracks costs for Azure Document Intelligence OCR:

response = litellm.ocr(
    model="azure_ai/doc-intelligence/prebuilt-layout",
    document={"type": "document_url", "document_url": "https://..."}
)

# Access cost information
print(f"Cost: ${response._hidden_params.get('response_cost', 0)}")

Additional Resources

• Azure Document Intelligence Documentation
• Pricing Details
• Supported File Formats
• LiteLLM OCR Documentation