Multi-Head Latent Attention (MLA)

A KV-cache compression technique for transformer attention, introduced in the DeepSeek-V2 paper and now the standard attention mechanism across DeepSeek V3, R1, Kimi K2, and the broader 2026 frontier-MoE ecosystem. Instead of sharing K/V tensors across query heads (the GQA approach), MLA performs **low-rank joint compression** of keys and values into a smaller latent space, storing only the compressed latent representation in the KV cache. At inference time the latent is projected back up to full dimensionality before computing attention.

Long-context inference is bottlenecked by KV-cache size. For a 128K-context model with standard Multi-Head Attention at DeepSeek-V3's dimensions, the KV cache alone would consume **~488 GB** — completely impractical for production serving. MLA stores a single length-512 latent per token, yielding a **64× smaller KV-cache footprint**. The trade-off historically was that compression hurts modeling quality, but DeepSeek demonstrated MLA achieves *better* modeling than standard MHA, which is why the DeepSeek team picked MLA over GQA and why every major open-weight frontier model since has adopted it.

Long-context inference where the KV-cache size dominates memory budget (128K+ contexts), agentic workloads that need to retain conversation state across many turns, production serving of MoE models where per-request memory matters more than tokens-per-second, retrofitting older transformer-based LLMs to support long context without retraining (per TransMLA), and any deployment where the memory savings justify the architectural complexity.