TurboQuant
Near-optimal KV-cache quantization technique from Google Research, published as the ICLR 2026 paper *arXiv 2504.19874*. Combines **PolarQuant** (random-rotation-matrix-then-optimal-scalar-quantization) with **QJL** (Quantized Johnson-Lindenstrauss) into a pipeline that compresses KV cache vectors with mathematically provable near-optimal distortion. **Data-oblivious** — requires no calibration data, no fine-tuning, no per-model setup. Empirical numbers: **6× KV cache memory reduction, up to 8× attention compute speedup** vs 32-bit keys, with **100% recall on Needle-In-A-Haystack up to 104,000 tokens**.
Definition
Near-optimal KV-cache quantization technique from Google Research, published as the ICLR 2026 paper *arXiv 2504.19874*. Combines **PolarQuant** (random-rotation-matrix-then-optimal-scalar-quantization) with **QJL** (Quantized Johnson-Lindenstrauss) into a pipeline that compresses KV cache vectors with mathematically provable near-optimal distortion. **Data-oblivious** — requires no calibration data, no fine-tuning, no per-model setup. Empirical numbers: **6× KV cache memory reduction, up to 8× attention compute speedup** vs 32-bit keys, with **100% recall on Needle-In-A-Haystack up to 104,000 tokens**.
Even after MLA's 64× KV-cache footprint reduction, serving long-context LLMs at scale remains memory-bound. Per-token KV vectors dominate VRAM in production inference, and the existing quantization options (INT8 / INT4 per-channel, AWQ, GPTQ-derived methods) all require some form of calibration data and tend to degrade quality on out-of-distribution prompts. TurboQuant's bet: a data-oblivious technique with provable distortion bounds (3-bit keys + 2-bit values) can be safer for production deployment than calibration-dependent quantization, because there's no "this calibration set drifted from prod" failure mode.
Production LLM inference serving where KV cache dominates VRAM, multi-tenant serving where calibration-per-tenant is operationally impractical, extending context windows by 6× on existing hardware without re-deploying larger GPUs, vLLM / SGLang / TensorRT-LLM forks experimenting with low-bit KV before official upstream support, and as a building block for the next generation of long-context serving stacks.
Recent developments
Google Research paper at ICLR 2026 (arXiv 2504.19874). Formal publication of the PolarQuant + QJL combination with provable distortion bounds. Per Google's TurboQuant — Nerd Level Tech.
First open-source implementation by OnlyTerp. First public open-source implementation of TurboQuant with 5× compression, near-zero quality loss. Per GitHub (OnlyTerp/turboquant).
vLLM fork with TurboQuant support (varjoranta/turboquant-vllm). Fork of vLLM 0.18.1rc1 adding TurboQuant+ KV cache compression: 3.8× smaller KV cache, same conversation quality, fused CUDA kernels with automatic PyTorch fallback. Per GitHub (varjoranta/turboquant-vllm).
0xSero/turboquant: 3-bit keys / 2-bit values with Triton kernels + vLLM integration. Alternate implementation pushing the aggressive quantization profile (3-bit keys + 2-bit values) with Triton-based kernels and direct vLLM integration. Per GitHub (0xSero/turboquant).
Absent from major inference frameworks as of April 2026 — Google official Q2 2026. TurboQuant is absent from official vLLM, TensorRT-LLM, and SGLang as of April 2026; Google's official implementation expected around Q2 2026. An official vLLM feature request is open. Per Kaitchup — TurboQuant finally fast and widely available.
Empirical 100% NIAH recall up to 104K tokens. Empirical evaluations show flawless 100% recall on "Needle-In-A-Haystack" tests extending up to 104,000 tokens — vastly outperforming traditional eviction methods. Per Medium — KV Cache Revolution.
Full benchmark detail: LongBench 3.5-bit score of 50.06 vs. FP16 baseline 50.16; NIAH 0.997 at 4x compression up to 104K tokens. Suite (LongBench, NIAH, ZeroSCROLLS, RULER, L-Eval) run across Llama-3.1-8B, Gemma, and Mistral. TQ3 profile: 4.9x compression, 52 bytes/128 values, MSE 0.034; TQ4 profile: 3.8x compression, 68 bytes, MSE 0.009. Per Nerd Level Tech.
Concrete memory savings: a Qwen2.5-3B KV cache at 8K tokens drops from 289 MB to ~58 MB; a 32B model's KV cache at 128K context drops from ~30 GB to ~6 GB. Per DecodeTheFuture and NVIDIA developer forum (both tertiary).
TurboQuant+ extends the technique to Apple Silicon, compressing 3.8–6.4x via PolarQuant + Walsh-Hadamard rotation, hitting q8_0 prefill-speed parity and 0.9x decode throughput at 128K context on an M5 Max, validated on 1.5B–104B models. Per ToolHunter (tertiary).
Pushback from an independent reimplementation: an MSE-only variant beat MSE+QJL for attention quality, and standard Q4_0 quantization beat TurboQuant at matched compression ratios. Asymmetric K/V bit allocation was found essential to TurboQuant's claimed advantage. Per tonbistudio/turboquant-pytorch (secondary, community finding — worth keeping in the node as a counterweight to the vendor benchmarks). Sources: Nerd Level Tech — Google's TurboQuant · GitHub (OnlyTerp/turboquant) · GitHub (varjoranta/turboquant-vllm) · GitHub (0xSero/turboquant) · Kaitchup — TurboQuant fast and widely available · Spheron Blog — 6× KV Cache Compression
Connections 4
Outbound 4
scoped_to1solves1enables1alternative_to1