LMCache
A high-performance distributed **KV-cache offloading** layer for LLM inference, written to maximize prefix-reuse across vLLM and other inference engines. Repository: [github.com/LMCache/LMCache](https://github.com/LMCache/LMCache). LMCache intercepts prefix tokens during prefill, persists their computed KV tensors to a distributed hierarchy (CPU memory → local NVMe → S3-compatible object storage), and serves them back instantly when the same prefix recurs — dramatically lowering Time-to-First-Token for repeated long-context queries. The **L2 Serde components** explicitly support S3 backends for datacenter-wide KV-cache persistence.
Definition
A high-performance distributed **KV-cache offloading** layer for LLM inference, written to maximize prefix-reuse across vLLM and other inference engines. Repository: [github.com/LMCache/LMCache](https://github.com/LMCache/LMCache). LMCache intercepts prefix tokens during prefill, persists their computed KV tensors to a distributed hierarchy (CPU memory → local NVMe → S3-compatible object storage), and serves them back instantly when the same prefix recurs — dramatically lowering Time-to-First-Token for repeated long-context queries. The **L2 Serde components** explicitly support S3 backends for datacenter-wide KV-cache persistence.
As prompts grow to millions of tokens, the "prefill tax" — the compute required to process the input sequence before generating the first token — becomes prohibitive. Moving a single bit of data through the memory hierarchy costs an order of magnitude more energy than the equivalent computation, so caching computed KV tensors is fundamentally more efficient than recomputing them. LMCache makes that cache durable, distributed, and S3-resident so it's amortized across the entire inference fleet.
Disaggregated prefill architectures, multi-node P2P CPU memory sharing for KV cache, persistent KV-cache pools backed by S3, prefix-reuse acceleration for vLLM deployments, CoreWeave-Cohere-style enterprise LLM serving.