Technology

LMCache

A high-performance distributed **KV-cache offloading** layer for LLM inference, written to maximize prefix-reuse across vLLM and other inference engines. Repository: [github.com/LMCache/LMCache](https://github.com/LMCache/LMCache). LMCache intercepts prefix tokens during prefill, persists their computed KV tensors to a distributed hierarchy (CPU memory → local NVMe → S3-compatible object storage), and serves them back instantly when the same prefix recurs — dramatically lowering Time-to-First-Token for repeated long-context queries. The **L2 Serde components** explicitly support S3 backends for datacenter-wide KV-cache persistence.

10 connections7 posts

Definition

What it is

A high-performance distributed **KV-cache offloading** layer for LLM inference, written to maximize prefix-reuse across vLLM and other inference engines. Repository: [github.com/LMCache/LMCache](https://github.com/LMCache/LMCache). LMCache intercepts prefix tokens during prefill, persists their computed KV tensors to a distributed hierarchy (CPU memory → local NVMe → S3-compatible object storage), and serves them back instantly when the same prefix recurs — dramatically lowering Time-to-First-Token for repeated long-context queries. The **L2 Serde components** explicitly support S3 backends for datacenter-wide KV-cache persistence.

Why it exists

As prompts grow to millions of tokens, the "prefill tax" — the compute required to process the input sequence before generating the first token — becomes prohibitive. Moving a single bit of data through the memory hierarchy costs an order of magnitude more energy than the equivalent computation, so caching computed KV tensors is fundamentally more efficient than recomputing them. LMCache makes that cache durable, distributed, and S3-resident so it's amortized across the entire inference fleet.

Primary use cases

Disaggregated prefill architectures, multi-node P2P CPU memory sharing for KV cache, persistent KV-cache pools backed by S3, prefix-reuse acceleration for vLLM deployments, CoreWeave-Cohere-style enterprise LLM serving.

Recent developments

Latest signals

  • Latest release: v0.5.5 (verified 2026-09-23). The 0.5.x line is the stable series, superseding 0.4.x. Per LMCache/LMCache releases.

  • LMCache paper at arXiv 2510.09665 formalized the enterprise KV-cache layer. "An Efficient KV Cache Layer for Enterprise-Scale LLM Inference" — academic framing for the production-cache patterns LMCache has shipped. First-class reference for the engineering community. Per arXiv 2510.09665 — LMCache: Efficient KV Cache Layer for Enterprise-Scale LLM Inference.

  • Now supports both vLLM AND SGLang. Extended beyond vLLM to also intercept and persist KV caches from SGLang — making LMCache the cross-engine KV-cache substrate for the two leading open-source inference engines. Per arXiv 2510.09665.

  • NetApp ONTAP S3 is the canonical shared-storage backend. NetApp published an end-to-end deployment guide for vLLM + LMCache + ONTAP S3, demonstrating both engines hitting the same offloaded KV-cache entries via S3. First major enterprise-storage vendor to ship a reference architecture. Per NetApp — KV Cache Offloading with vLLM, LMCache, and ONTAP S3.

  • vLLM production-stack ships LMCache as the canonical KV-cache offloader. vllm-project/production-stack includes LMCache + the KV-cache-offloading tutorial as a first-class deployment pattern. Confirms LMCache as the reference for "vLLM + persistent KV cache." Per GitHub — vllm-project/production-stack KV cache offload tutorial.

  • KServe also supports LMCache for Hugging Face vLLM backends. Kubernetes-native model-serving (KServe) added LMCache integration for the HF vLLM runtime — KV cache offloading now sits inside the standard K8s ML-serving stack. Per KServe — KV Cache Offloading with Huggingface vLLM Backend.

  • Tiered storage: CPU memory → local NVMe → S3 — the "energy moves more than data" framing. Production deployments use the full tiered hierarchy: moving a single bit through memory costs an order of magnitude more energy than recomputing it locally — so caching tiers down to S3 amortizes that energy across the entire fleet. Per NetApp — Engineering Inference: KV Cache, Shared Storage, Economics of AI.

  • The break-even math for S3-tier KV caching now has a number: roughly one cache reuse every 94 hours. Tensormesh's vendor-published ROI model prices Tier-2 S3-compatible storage for activation tensors at ~$4,000–$5,000 per TB over three years; assuming a 10× speedup for a cache hit versus recomputing tokens, ~280 hits over that period — one every ~94 hours — justifies the storage spend. Vendor-published methodology, but it's the first public framework that turns "should we persist KV cache to object storage" from intuition into arithmetic. Per LMCache Storage ROI Calculator (Tensormesh).

  • Production TTFT evidence: Cohere on LMCache + CoreWeave AI Object Storage. The published case study reports 22–32% lower time-to-first-token and 41% higher decode throughput versus full recomputation, with S3-tier cache retrievals 1.2× faster than AWS S3 Express One Zone when cold and up to 3× faster when hot — the first named-customer numbers for KV-cache-on-object-storage serving. Per LMCache blog — Cohere case study.

  • Tensormesh's commercial layer publishes retrieval-SLO engineering numbers. CacheBlend non-prefix caching claims up to 25× better hit rates for agent workloads by matching contiguous token blocks regardless of prompt position, and a Redis-backend rework reaches 9–10 GB/s KV-chunk retrieval over ConnectX-7 NICs (30× standard clients) against a ~2 GB/s retrieval SLO. Vendor-published. Per Tensormesh — non-prefix caching and Tensormesh — Redis KV-cache optimization.

  • Second named S3-tier KV-cache deployment: Dell ObjectScale's GPUDirect RDMA offload path merged upstream into vLLM and LMCache (September 2026). Dell integrated GPUDirect RDMA KV-cache offloading into ObjectScale, its S3-compatible platform, with the capability merged upstream into both vLLM and LMCache — supporting S3-HTTP and S3-over-RDMA paths (vendor-published release notes). This is the corroboration our dated 2026-08-28 call was waiting on directionally: a second major vendor treating object storage as a disaggregated, durable memory tier, alongside the CoreWeave/Cohere deployment, moves the pattern from single-anecdote toward standardized enterprise architecture. Latency numbers for the Dell path have not yet been published — the call's clause (a) wants those before it closes. Per Dell — KV cache offload to object storage: GPU-Direct and upstream.

  • Google Cloud GKE Inference joins the named production-user roster (September 2026). The project's current enterprise framing lists Google Cloud GKE Inference, CoreWeave, and Cohere as production users since LMCache's January 2026 production graduation — a hyperscaler-managed inference platform is a materially bigger adoption signal than a single named customer, though the GKE deployment is not specifically documented as using the S3 storage tier, and no independent replication of the Cohere TTFT numbers has been published. Adoption breadth is growing; independent corroboration is still owed. Per LMCache paper v2 (arXiv 2510.09665) and LMCache GitHub (project-published). Sources: arXiv 2510.09665 — LMCache: Efficient KV Cache for Enterprise-Scale LLM Inference · LMCache — Offload KV Cache docs · NetApp — vLLM + LMCache + ONTAP S3 · vLLM production-stack — KV cache offload tutorial · KServe — KV Cache Offloading with Huggingface vLLM

Connections10

Outbound5
scoped_to2
AI Memory InfrastructureS3
integrates_with1
vLLM
stores1
AWS S3
optimizes_for1
Prefill Tax
Inbound5
integrates_with3
Dell ObjectScalevLLMCacheGen
depends_on2
KV-Cache DisaggregationHierarchical KV Cache Architecture

Featured in

2026-09-26When the Catalog Started Handing Out KeysSeptember 2026 turned the lakehouse catalog into an agent-callable control plane and, in the same motion, into the thing holding your object-storage credentials. OpenMetadata 2.0 ships MCP on by default and rebrands as a context layer. Unity Catalog 0.6.0 quietly expires tokens that were previously valid forever and re-keys its credential cache. And CVE-2026-34214 showed what the failure mode looks like in production: a query engine handing bucket keys to anyone with SQL write privilege. Three facts, one boundary.2026-09-03When the Network Became the Storage BottleneckThe pain point this index mapped in May — GPUs idle, waiting on object storage — just got its industry answer, and it isn't faster storage software. It's three competing network fabrics: Ultra Ethernet 1.0 with packet spraying, InfiniBand XDR at 800G per port, and RoCEv2 eating the middle. Every S3-to-GPU byte now crosses a wire with three possible futures, and the storage tier's ceiling is set by which one you pick.2026-08-28The Half-Life of a NumberFour days after we published the storage-tier thesis, a verification sweep caught two of our own numbers decaying — one because the paper behind it quietly revised itself, one because a viral figure never matched the audited price trail. Both are corrected. The same sweep delivered the production evidence our dated call was waiting for, which is the uncomfortable part: verification and vindication arrived in the same run.2026-08-24The Storage Tier Is the New Context WindowA vendor published the break-even arithmetic for persisting LLM inference state to object storage: one cache reuse every 94 hours justifies the spend. Between that number, NVMe object tiers priced like commodities, and streaming engines that write payloads straight to S3, the quiet reclassification of object storage — from cold archive to active memory tier — now has receipts.2026-07-26When the Benchmarks Stopped Agreeing: Agent Memory's Trust Crisis Meets the Priced CacheThe same memory system scores 92.5% or 55% depending on who runs the benchmark. Meanwhile cached input tokens are being billed at zero and cache-hit-rate became a production SLO. July 2026 is the month agent memory's two crises — trust and cost — turned out to be the same story, and the seam between them is where the next incident lives.2026-07-21Memory Became a Market: The AI Data Stack is Being Rebuilt on Three FrontsIn one week of July 2026: a benchmark war between agent-memory vendors with irreconcilable numbers, a second critical CVE in the most popular memory layer, a columnar format field fracturing into workload-specific successors to Parquet, and 3-bit KV-cache quantization getting its first independent reproduction. Underneath all three fronts, a runaway enterprise SSD price surge is forcing the whole stack onto object storage. The rebuild isn't coming — it's live.2026-05-16When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory TierStateful agents needed stateful storage. Tonight 33 new nodes joined the index: Mem0, Zep, LMCache, SGLang, Mooncake, MCP, cuObject, BlueField-4, Animesis CMA. The architectural thesis: AI memory infrastructure crystallized around object storage in 2026, and the persistence layer the industry settled on is S3.