---
title: "Gradio Chatbot + LiteLLM Tutorial"
description: "Credits to [ZQ](https://x.com/ZQ_Dev), for this tutorial."
type: skill
canonical_url: https://claudary.paisolsolutions.com/skills/gradio-integration
source: "Claudary"
difficulty: intermediate
author: "Claude Code Knowledge Pack"
date: 2026-07-10T11:25:03.397Z
license: CC-BY-4.0
attribution: "Gradio Chatbot + LiteLLM Tutorial — Claudary (https://claudary.paisolsolutions.com/skills/gradio-integration)"
---

# Gradio Chatbot + LiteLLM Tutorial
Credits to [ZQ](https://x.com/ZQ_Dev), for this tutorial.

## Overview

# Gradio Chatbot + LiteLLM Tutorial
Simple tutorial for integrating LiteLLM completion calls with streaming Gradio chatbot demos

### Install & Import Dependencies
```python
!uv add gradio litellm
import gradio
import litellm
```

### Define Inference Function
Remember to set `model` and `api_base` as expected by the server hosting your LLM.
```python
def inference(message, history):
    try:
        flattened_history = [item for sublist in history for item in sublist]
        full_message = " ".join(flattened_history + [message])
        messages_litellm = [{"role": "user", "content": full_message}] # litellm message format
        partial_message = ""
        for chunk in litellm.completion(model="huggingface/meta-llama/Llama-2-7b-chat-hf",
                                        api_base="x.x.x.x:xxxx",
                                        messages=messages_litellm,
                                        max_new_tokens=512,
                                        temperature=.7,
                                        top_k=100,
                                        top_p=.9,
                                        repetition_penalty=1.18,
                                        stream=True):
            partial_message += chunk['choices'][0]['delta']['content'] # extract text from streamed litellm chunks
            yield partial_message
    except Exception as e:
        print("Exception encountered:", str(e))
        yield f"An Error occurred please 'Clear' the error and try your question again"
```

### Define Chat Interface
```python
gr.ChatInterface(
    inference,
    chatbot=gr.Chatbot(height=400),
    textbox=gr.Textbox(placeholder="Enter text here...", container=False, scale=5),
    description=f"""
    CURRENT PROMPT TEMPLATE: {model_name}.
    An incorrect prompt template will cause performance to suffer.
    Check the API specifications to ensure this format matches the target LLM.""",
    title="Simple Chatbot Test Application",
    examples=["Define 'deep learning' in once sentence."],
    retry_btn="Retry",
    undo_btn="Undo",
    clear_btn="Clear",
    theme=theme,
).queue().launch()
```
### Launch Gradio App
1. From command line: `python app.py` or `gradio app.py` (latter enables live deployment updates)
2. Visit provided hyperlink in your browser.
3. Enjoy prompt-agnostic interaction with remote LLM server.

### Recommended Extensions:
* Add command line arguments to define target model & inference endpoints

Credits to [ZQ](https://x.com/ZQ_Dev), for this tutorial.

---

Source: [Claudary](https://claudary.paisolsolutions.com/skills/gradio-integration) · https://claudary.paisolsolutions.com