LiteLLM

An open-source **model gateway** that abstracts the complexity of calling hundreds of different LLM endpoints behind a unified, OpenAI-compatible API. Provides load balancing, automatic failover, cost optimization, rate limiting, and — critically for S3-relevance — **semantic-cache backends targeting S3** (`type: s3` in the LiteLLM config schema). When a query hits the gateway with a high-confidence semantic match against a cached prompt, LiteLLM returns the cached response instantly, bypassing the upstream LLM provider and the associated per-token cost.

Production LLM deployments rarely commit to a single model vendor — costs shift, capability windows move, regional availability varies, and failover requirements force multi-provider strategies. LiteLLM provides a single API surface that hides the multi-vendor complexity, plus an S3-backed semantic cache that converts repetitive agentic queries into near-zero-cost lookups. The S3 cache layer is the operationally interesting piece: it survives across gateway restarts and is shared across all gateway instances behind a load balancer.

Multi-LLM-provider serving with unified API, S3-backed semantic prompt caching for cost optimization, automatic failover across model vendors, audit-log streaming to S3 for observability and compliance, gateway-as-rate-limiter for cost governance.