Prefill-Decode Disaggregation

An LLM-serving architecture pattern that splits the two compute phases of transformer inference — **prefill** (compute-bound, processes the entire prompt in one forward pass to fill the KV-cache) and **decode** (memory-bandwidth-bound, generates one token per pass over the existing KV-cache) — into separate worker pools, each optimized for its phase. The completed KV-cache is shipped from prefill workers to decode workers via RDMA, NVLink, or (with CacheGen-style compression) commodity Ethernet.

A single GPU pool serving both prefill and decode is fundamentally unbalanced. Prefill saturates compute but barely touches HBM bandwidth; decode saturates HBM bandwidth but underutilizes compute. The two phases interfere with each other under shared scheduling — prefill batches starve decode tail latency, decode batches block prefill throughput. Disaggregation lets each pool run at peak utilization, and lets the operator buy *different* hardware for each (compute-heavy B200 for prefill, memory-bandwidth-heavy H100 for decode).

High-throughput production LLM serving at scale (any deployment ≥4 GPUs benefits), multi-tenant platforms with mixed prompt-length distributions, long-context serving where prefill amortization matters most, agentic workflows where prefix-cache hits make prefill skippable (further leverage from disaggregation).