KV-Cache Disaggregation

An architectural pattern that decouples LLM inference compute from inference **state** (the KV-cache), enabling that state to be stored in tiered, network-attached memory (CPU DRAM, CXL DRAM, NVMe, S3-compatible object storage) rather than living solely in GPU HBM. KV-Cache Disaggregation encompasses three sub-patterns: prefill/decode pool separation (compute-phase disaggregation), tiered KV-cache memory (storage-tier disaggregation), and cross-node prefix-cache federation (geographic disaggregation).

GPU HBM is the scarcest, most expensive memory tier in any LLM-serving stack. As context windows extended from 4k to 1M+ tokens and multi-agent workflows generated massive volumes of derived state, KV-cache demand exceeded HBM capacity by 10-100x. Storing all KV-cache in HBM is economically infeasible; storing it only in CPU DRAM kills decode throughput. Disaggregation is the structural fix: HBM holds hot working set, DRAM holds warm prefix caches, NVMe holds long-tail prefixes, S3-compatible object storage holds cross-session episodic context. The disaggregation pattern makes the economics of long-context serving work.