Technology

LiteLLM

An open-source **model gateway** that abstracts the complexity of calling hundreds of different LLM endpoints behind a unified, OpenAI-compatible API. Provides load balancing, automatic failover, cost optimization, rate limiting, and — critically for S3-relevance — **semantic-cache backends targeting S3** (`type: s3` in the LiteLLM config schema). When a query hits the gateway with a high-confidence semantic match against a cached prompt, LiteLLM returns the cached response instantly, bypassing the upstream LLM provider and the associated per-token cost.

10 connections5 posts

Definition

What it is

An open-source **model gateway** that abstracts the complexity of calling hundreds of different LLM endpoints behind a unified, OpenAI-compatible API. Provides load balancing, automatic failover, cost optimization, rate limiting, and — critically for S3-relevance — **semantic-cache backends targeting S3** (`type: s3` in the LiteLLM config schema). When a query hits the gateway with a high-confidence semantic match against a cached prompt, LiteLLM returns the cached response instantly, bypassing the upstream LLM provider and the associated per-token cost.

Why it exists

Production LLM deployments rarely commit to a single model vendor — costs shift, capability windows move, regional availability varies, and failover requirements force multi-provider strategies. LiteLLM provides a single API surface that hides the multi-vendor complexity, plus an S3-backed semantic cache that converts repetitive agentic queries into near-zero-cost lookups. The S3 cache layer is the operationally interesting piece: it survives across gateway restarts and is shared across all gateway instances behind a load balancer.

Primary use cases

Multi-LLM-provider serving with unified API, S3-backed semantic prompt caching for cost optimization, automatic failover across model vendors, audit-log streaming to S3 for observability and compliance, gateway-as-rate-limiter for cost governance.

Recent developments

Latest signals

  • Latest release: v1.97.0 (stable, August 16, 2026), succeeding v1.96.0. 1.97.0 drops conflicting tool choices for Bedrock and improves transport backoff handling. LiteLLM dropped the -stable version suffix at v1.84.0 and now ships plain semver; the legacy main-stable Docker tag is deprecated (retired June 30, 2026). Per BerriAI/litellm releases.

  • Supply-chain history to know: versions 1.82.7 and 1.82.8 were compromised on PyPI (March 2026). The TeamPCP supply-chain campaign published malicious LiteLLM releases via a compromised CI action — one of the highest-profile PyPI compromises of 2026 given LiteLLM's position in front of production LLM traffic. The 1.9x line is clean and the project remains highly active, but any environment that installed 1.82.7/1.82.8 during the exposure window should be treated as compromised and rotated. A concrete reminder that the LLM-router layer concentrates credentials for every provider behind it. Per Datadog Security Labs' TeamPCP analysis.

  • 140+ providers, 2,500+ models behind a single OpenAI-compatible interface (March 2026). Covers OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, Mistral, Ollama, vLLM, plus emerging providers like Nebius AI. The reference "one gateway to call them all" pattern for production LLM stacks. Per a2a-mcp — What Is LiteLLM? 140+ Providers 2026.

  • ~40K GitHub stars, 1,300+ contributors. Among the most-actively maintained AI infrastructure projects in 2026; rapid model additions as new providers launch. Per GitHub — BerriAI/litellm.

  • Agent Hub, MCP support, sidecar architecture added. 2026 LiteLLM releases extended the proxy with an Agent Hub (cross-team agent registry), Model Context Protocol integration, and a sidecar architecture that reduces proxy overhead for high-QPS deployments. Per Nerd Level Tech — LiteLLM Proxy Production Tutorial 2026.

  • Production stack: Docker Compose + Postgres + virtual keys + per-team budgets + auto-failover. Standard 2026 deployment shape: LiteLLM proxy with Postgres for analytics/spend tracking, virtual API keys per team, and automatic fallback routing across Claude Sonnet 4.6 / GPT-5.4 / Gemini 2.5 Pro. Per Nerd Level Tech — LiteLLM Proxy Production Tutorial 2026.

  • Multi-tenant features: per-project logging/guardrails/caching, virtual keys, admin UI. Production-grade governance layer means LiteLLM is increasingly chosen at the platform-team level, not just by individual app developers. Per LiteLLM — AI Gateway docs.

  • Maxim 2026 LLM-router benchmark places LiteLLM in the top tier. Alongside Portkey, Helicone, and Vercel AI Gateway — LiteLLM wins on provider count and OSS extensibility; commercial gateways win on managed-service polish. Per Maxim — Top 5 LLM Router Solutions 2026.

  • The March compromise now has a CVE and a wider blast radius than "two bad releases." CVE-2026-33634 (CVSS 9.4): the TeamPCP campaign first poisoned Aqua Trivy's GitHub Actions, harvested publishing credentials from CI pipelines running the compromised scanner, propagated the CanisterWorm across npm, hijacked Checkmarx KICS and Docker Hub images, and published LiteLLM 1.82.7/1.82.8 to PyPI on March 24, 2026. The payload writes litellm_init.pth into site-packages — executing on every Python interpreter start — so uninstalling LiteLLM does not disinfect a host. Remediation consensus: destroy affected environments, rotate all cloud and AI-provider keys, pin CI dependencies to immutable SHAs. Per Averlon — CVE-2026-33634, Trend Micro — inside the LiteLLM compromise, and Unit 42 — TeamPCP.

  • Final incident scope (September 2026 verification): 1,000+ enterprise SaaS environments affected; no new AI-toolchain compromises of this magnitude since late August. The blast radius traces to LiteLLM's position as a transitive dependency inside many AI gateways — the .pth persistence triggered an exponential fork loop (litellm_init.pth) on interpreter startup, driving RAM exhaustion and lateral-movement attempts across Kubernetes clusters (independent-methodology security analysis). The structural flaw stands regardless: poisoned CI/CD tooling can inject backdoors into AI-orchestration libraries that auto-execute at startup. Per Arctic Wolf — TeamPCP campaign scope and Cycode — LiteLLM supply-chain attack response. Sources: GitHub — BerriAI/litellm · LiteLLM — AI Gateway docs · a2a-mcp — What Is LiteLLM? 140+ Providers 2026 · Nerd Level Tech — LiteLLM Proxy Production Tutorial 2026 · Maxim — Top 5 LLM Router Solutions 2026

Connections10

Outbound7
scoped_to2
AI Runtime InfrastructureS3
implements1
S3 API
stores1
AWS S3
acts_as1
AI Runtime Infrastructure
solves2
High Cloud Inference CostAI Subscription Quota Opacity
Inbound3
competes_with2
Helicone AI GatewayTraefik AI Gateway
related_to1
AI Toolchain Supply-Chain Compromise

Featured in

2026-09-03When the Quota Became a VariableWe measured our own $20 inference plan dropping from ~2M tokens per session to ~700K — same plan, same price, no changelog. Then we pulled the industry-wide evidence: a documented 2025–2026 wave of quota re-denominations, peak-hour throttles admitted only after user telemetry caught them, and deprecations that silently rebill at 6× the rate. Also the counter-evidence, because the thesis only survives if it's tested: OpenAI uncapped text, Anthropic doubled limits in May, and DeepSeek's price floor keeps falling. The pattern isn't uniform gouging — it's bifurcation plus unit games. And it's the strongest argument yet for the stack this index maps.2026-09-03When the Network Became the Storage BottleneckThe pain point this index mapped in May — GPUs idle, waiting on object storage — just got its industry answer, and it isn't faster storage software. It's three competing network fabrics: Ultra Ethernet 1.0 with packet spraying, InfiniBand XDR at 800G per port, and RoCEv2 eating the middle. Every S3-to-GPU byte now crosses a wire with three possible futures, and the storage tier's ceiling is set by which one you pick.2026-08-28The Half-Life of a NumberFour days after we published the storage-tier thesis, a verification sweep caught two of our own numbers decaying — one because the paper behind it quietly revised itself, one because a viral figure never matched the audited price trail. Both are corrected. The same sweep delivered the production evidence our dated call was waiting for, which is the uncomfortable part: verification and vindication arrived in the same run.2026-08-24The Storage Tier Is the New Context WindowA vendor published the break-even arithmetic for persisting LLM inference state to object storage: one cache reuse every 94 hours justifies the spend. Between that number, NVMe object tiers priced like commodities, and streaming engines that write payloads straight to S3, the quiet reclassification of object storage — from cold archive to active memory tier — now has receipts.2026-05-16When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory TierStateful agents needed stateful storage. Tonight 33 new nodes joined the index: Mem0, Zep, LMCache, SGLang, Mooncake, MCP, cuObject, BlueField-4, Animesis CMA. The architectural thesis: AI memory infrastructure crystallized around object storage in 2026, and the persistence layer the industry settled on is S3.