Hierarchical KV Cache Architecture

A four-tier storage architecture for LLM-inference KV-cache, layering: **(L1)** active working-set KV-cache in GPU HBM; **(L2)** pinned CPU DRAM as a hot intermediary across the PCIe bus; **(L3)** local NVMe (often with GPUDirect Storage) for long-context payloads exceeding DRAM; **(L4)** remote/distributed tier — Mooncake's pooled cluster DRAM+NVMe or S3-compatible object storage — as the durable, globally accessible repository. The architecture pairs with chunked-prefetch logic that exploits inference-queue idle intervals to stage required prefix-caches from L3/L4 up to L1/L2 *before* the compute step demands them.

Single-tier KV-cache management forces an unwinnable trade-off — keep everything in HBM (capacity hits the wall at ~10s of GB per accelerator), or fetch on demand from slower tiers (decode-stage latency explodes). The hierarchical pattern lets each tier do what it's good at: HBM serves the immediate decode step, DRAM serves the next-1-second window, NVMe serves the next-1-minute window, S3 serves the multi-session durable archive. With chunked prefetch hiding the L4→L1 staging behind GPU idle time, the architecture delivers near-HBM-latency reads for cache sizes orders of magnitude larger than HBM capacity.