--- title: "index 73" description: "import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import VersionVerificationTable from '@site/src/components/VersionVerificationTable';" type: skill canonical_url: https://claudary.paisolsolutions.com/skills/index-73 source: "Claudary" difficulty: intermediate author: "Claude Code Knowledge Pack" date: 2026-07-10T11:25:31.017Z license: CC-BY-4.0 attribution: "index 73 — Claudary (https://claudary.paisolsolutions.com/skills/index-73)" --- # index 73 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import VersionVerificationTable from '@site/src/components/VersionVerificationTable'; ## Overview --- slug: security-update-march-2026 title: "Security Update: Suspected Supply Chain Incident" date: 2026-03-24T14:00:00 authors: - krrish - ishaan-alt description: "As of 2:00 PM ET on March 24, 2026" tags: [security, incident-report] hide_table_of_contents: false --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import VersionVerificationTable from '@site/src/components/VersionVerificationTable'; > **Status:** Active investigation > **Last updated:** March 27, 2026 > **Update (March 30):** A new **clean** version of LiteLLM is now available (v1.83.0). This was released by our new [CI/CD v2](https://docs.litellm.ai/blog/ci-cd-v2-improvements) pipeline which added isolated environments, stronger security gates, and safer release separation for LiteLLM. > **Update (March 27):** Review Townhall updates, including explanation of the incident, what we've done, and what comes next. [Learn more](https://docs.litellm.ai/blog/security-townhall-updates) > **Update (March 27):** Added [Verified safe versions](#verified-safe-versions) section with SHA-256 checksums for all audited PyPI and Docker releases. > **Update (March 26):** Added `checkmarx[.]zone` to [Indicators of compromise](#indicators-of-compromise-iocs) > **Update (March 25):** Added community-contributed scripts for scanning GitHub Actions and GitLab CI pipelines for the compromised versions. See [How to check if you are affected](#how-to-check-if-you-are-affected). s/o [@Zach Fury](https://www.linkedin.com/in/fryware/) for these scripts. ## TLDR; - The compromised PyPI packages were **litellm==1.82.7** and **litellm==1.82.8**. Those packages were live on March 24, 2026 from 10:39 UTC for about 40 minutes before being quarantined by PyPI. - We believe that the compromise originated from the [Trivy dependency](https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/) used in our CI/CD security scanning workflow. - Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. - ~~We have paused all new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.~~ **Updated:** We have now released a new **safe** version of LiteLLM (v1.83.0) by our new [CI/CD v2](https://docs.litellm.ai/blog/ci-cd-v2-improvements) pipeline which added isolated environments, stronger security gates, and safer release separation for LiteLLM. We have also verified the codebase is safe and no malicious code was pushed to `main`. ## Overview LiteLLM AI Gateway is investigating a suspected supply chain attack involving unauthorized PyPI package publishes. Current evidence suggests a maintainer's PyPI account may have been compromised and used to distribute malicious code. At this time, we believe this incident may be linked to the broader [Trivy security compromise](https://www.aquasec.com/blog/trivy-supply-chain-attack-what-you-need-to-know/), in which stolen credentials were reportedly used to gain unauthorized access to the LiteLLM publishing pipeline. This investigation is ongoing. Details below may change as we confirm additional findings. ## Confirmed affected versions The following LiteLLM versions published to PyPI were impacted: - **v1.82.7**: contained a malicious payload in the LiteLLM AI Gateway `proxy_server.py` - **v1.82.8**: contained `litellm_init.pth` and a malicious payload in the LiteLLM AI Gateway `proxy_server.py` If you installed or ran either of these versions, review the recommendations below immediately. Note: These versions have already been removed from PyPI. ## What happened Initial evidence suggests the attacker bypassed official CI/CD workflows and uploaded malicious packages directly to PyPI. These compromised versions appear to have included a credential stealer designed to: - Harvest secrets by scanning for: - environment variables - SSH keys - cloud provider credentials (AWS, GCP, Azure) - Kubernetes tokens - database passwords - Encrypt and exfiltrate data via a `POST` request to `models.litellm.cloud`, which is **not** an official BerriAI / LiteLLM domain ## Who is affected You may be affected if **any** of the following are true: - You installed or upgraded LiteLLM via `pip` on **March 24, 2026**, between **10:39 UTC and 16:00 UTC** - You ran `pip install litellm` without pinning a version and received **v1.82.7** or **v1.82.8** - You built a Docker image during this window that included `pip install litellm` without a pinned version - A dependency in your project pulled in LiteLLM as a transitive, unpinned dependency (for example through AI agent frameworks, MCP servers, or LLM orchestration tools) You are **not** affected if any of the following are true: **LiteLLM AI Gateway/Proxy users:** Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. - You are using **LiteLLM Cloud** - You are using the official LiteLLM AI Gateway Docker image: `ghcr.io/berriai/litellm` - You are on **v1.82.6 or earlier** and did not upgrade during the affected window - You installed LiteLLM from source via the GitHub repository, which was **not** compromised ### How to check if you are affected ```bash pip show litellm ``` Go to the proxy base url, and check the version of the installed LiteLLM. ![Proxy version check](../../img/security_update_march_2026/proxy_version.png) Scans all repositories in a GitHub organization for workflow jobs that installed the compromised versions. **Requirements:** Python 3 and `requests` (`pip install requests`). **Setup:** ```bash export GITHUB_TOKEN="your-github-pat" ``` **Run:** ```bash python find_litellm_github.py ``` Set the `ORG` variable in the script to your GitHub organization name. Both scripts default to scanning jobs from **today**. Adjust the `WINDOW_START` and `WINDOW_END` constants to cover **March 24, 2026** (the incident date) if running on a different day.
View full script (find_litellm_github.py) ```python #!/usr/bin/env python3 """ Scan all GitHub Actions jobs in a GitHub org that ran between 0800-1244 UTC today and identify any that installed litellm 1.82.7 or 1.82.8. Adjust WINDOW_START / WINDOW_END to cover March 24, 2026 if running later. """ import io import os import re import sys import zipfile from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime, timezone import requests GITHUB_URL = "https://api.github.com" ORG = "your-org" # <-- set to your GitHub organization TOKEN = os.environ.get("GITHUB_TOKEN", "") TODAY = datetime.now(timezone.utc).date() WINDOW_START = datetime(TODAY.year, TODAY.month, TODAY.day, 8, 0, 0, tzinfo=timezone.utc) WINDOW_END = datetime(TODAY.year, TODAY.month, TODAY.day, 12, 44, 0, tzinfo=timezone.utc) TARGET_VERSIONS = {"1.82.7", "1.82.8"} VERSION_PATTERN = re.compile(r"litellm[=\\-](\\d+\\.\\d+\\.\\d+)", re.IGNORECASE) SESSION = requests.Session() SESSION.headers.update({ "Authorization": f"Bearer {TOKEN}", "Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28", }) def get_paginated(url, params=None): params = dict(params or {}) params.setdefault("per_page", 100) page = 1 while True: params["page"] = page resp = SESSION.get(url, params=params, timeout=30) if resp.status_code == 404: return resp.raise_for_status() data = resp.json() if isinstance(data, dict): items = next((v for v in data.values() if isinstance(v, list)), []) else: items = data if not items: break yield from items if len(items) < params["per_page"]: break page += 1 def parse_ts(ts_str): if not ts_str: return None return datetime.fromisoformat(ts_str.replace("Z", "+00:00")) def get_repos(): repos = [] for r in get_paginated(f"{GITHUB_URL}/orgs/{ORG}/repos", {"type": "all"}): repos.append({"id": r["id"], "name": r["name"], "full_name": r["full_name"]}) return repos def get_runs_in_window(repo_full_name): created_filter = ( f"{WINDOW_START.strftime('%Y-%m-%dT%H:%M:%SZ')}" f"..{WINDOW_END.strftime('%Y-%m-%dT%H:%M:%SZ')}" ) url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/runs" runs = [] for run in get_paginated(url, {"created": created_filter, "per_page": 100}): ts = parse_ts(run.get("run_started_at") or run.get("created_at")) if ts and WINDOW_START <= ts <= WINDOW_END: runs.append(run) return runs def get_jobs_for_run(repo_full_name, run_id): url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/runs/{run_id}/jobs" jobs = [] for job in get_paginated(url, {"filter": "all"}): ts = parse_ts(job.get("started_at")) if ts and WINDOW_START <= ts <= WINDOW_END: jobs.append(job) return jobs def fetch_job_log(repo_full_name, job_id): url = f"{GITHUB_URL}/repos/{repo_full_name}/actions/jobs/{job_id}/logs" resp = SESSION.get(url, timeout=60, allow_redirects=True) if resp.status_code in (403, 404, 410): return "" resp.raise_for_status() content_type = resp.headers.get("Content-Type", "") if "zip" in content_type or resp.content[:2] == b"PK": try: with zipfile.ZipFile(io.BytesIO(resp.content)) as zf: parts = [] for name in sorted(zf.namelist()): with zf.open(name) as f: parts.append(f.read().decode("utf-8", errors="replace")) return "\\n".join(parts) except zipfile.BadZipFile: pass return resp.text def check_job(repo_full_name, job): job_id = job["id"] job_name = job["name"] run_id = job["run_id"] started = job.get("started_at", "") log_text = fetch_job_log(repo_full_name, job_id) if not log_text: return None found_versions = set() context_lines = [] for line in log_text.splitlines(): m = VERSION_PATTERN.search(line) if m: ver = m.group(1) if ver in TARGET_VERSIONS: found_versions.add(ver) context_lines.append(line.strip()) if not found_versions: return None return { "repo": repo_full_name, "run_id": run_id, "job_id": job_id, "job_name": job_name, "started_at": started, "versions": sorted(found_versions), "context": context_lines[:10], "job_url": job.get("html_url", f"https://github.com/{repo_full_name}/actions/runs/{run_id}"), } def main(): if not TOKEN: print("ERROR: Set GITHUB_TOKEN environment variable.", file=sys.stderr) sys.exit(1) print(f"Time window : {WINDOW_START.isoformat()} -> {WINDOW_END.isoformat()}") print(f"Hunting for : litellm {', '.join(sorted(TARGET_VERSIONS))}") print() print(f"Fetching repositories for org '{ORG}'...") repos = get_repos() print(f" Found {len(repos)} repositories") print() jobs_to_check = [] print("Scanning workflow runs for time window...") for repo in repos: full_name = repo["full_name"] try: runs = get_runs_in_window(full_name) except requests.HTTPError as e: print(f" WARN: {full_name} - {e}", file=sys.stderr) continue if not runs: continue print(f" {full_name}: {len(runs)} run(s) in window") for run in runs: try: jobs = get_jobs_for_run(full_name, run["id"]) except requests.HTTPError as e: print(f" WARN: run {run['id']} - {e}", file=sys.stderr) continue for job in jobs: jobs_to_check.append((full_name, job)) total = len(jobs_to_check) print(f"\\nFetching logs for {total} job(s)...") print() hits = [] with ThreadPoolExecutor(max_workers=8) as pool: futures = { pool.submit(check_job, full_name, job): (full_name, job["id"]) for full_name, job in jobs_to_check } done = 0 for future in as_completed(futures): done += 1 full_name, jid = futures[future] try: result = future.result() except Exception as e: print(f" ERROR {full_name} job {jid}: {e}", file=sys.stderr) continue if result: hits.append(result) print( f" [{done}/{total}] {full_name} job {jid}" + (f" *** HIT: litellm {result['versions']} ***" if result else ""), flush=True, ) print() print("=" * 72) print(f"RESULTS: {len(hits)} job(s) installed litellm {' or '.join(sorted(TARGET_VERSIONS))}") print("=" * 72) if not hits: print("No matches found.") return for h in sorted(hits, key=lambda x: x["started_at"]): print() print(f" Repo : {h['repo']}") print(f" Job : {h['job_name']} (#{h['job_id']})") print(f" Run ID : {h['run_id']}") print(f" Started : {h['started_at']}") print(f" Versions : litellm {', '.join(h['versions'])}") print(f" URL : {h['job_url']}") print(f" Log lines :") for line in h["context"]: print(f" {line}") if __name__ == "__main__": main() ```
Scans all projects in a GitLab group (including subgroups) for CI/CD jobs that installed the compromised versions. **Requirements:** Python 3 and `requests` (`pip install requests`). **Setup:** ```bash export GITLAB_TOKEN="your-gitlab-pat" ``` **Run:** ```bash python find_litellm_jobs.py ``` Set the `GROUP_NAME` variable in the script to your GitLab group name. Both scripts default to scanning jobs from **today**. Adjust the `WINDOW_START` and `WINDOW_END` constants to cover **March 24, 2026** (the incident date) if running on a different day.
View full script (find_litellm_jobs.py) ```python #!/usr/bin/env python3 """ Scan all GitLab CI/CD jobs in a GitLab group that ran between 0800-1244 UTC today and identify any that installed litellm 1.82.7 or 1.82.8. Adjust WINDOW_START / WINDOW_END to cover March 24, 2026 if running later. """ import os import re import sys from concurrent.futures import Th --- Source: [Claudary](https://claudary.paisolsolutions.com/skills/index-73) · https://claudary.paisolsolutions.com