Technology

Kafka Tiered Storage

An Apache Kafka feature (KIP-405) that offloads older log segments from broker-local disks to S3-compatible object storage, extending Kafka's retention capacity without scaling broker storage proportionally.

10 connections3 resources

Summary

What it is

Where it fits

Kafka Tiered Storage bridges the gap between real-time event streaming and long-term S3 storage. By transparently moving cold log segments to S3, it allows Kafka to serve as both the streaming platform and a long-retention event archive, reducing the need for separate S3 sink connectors for archival.

Misconceptions / Traps

Tiered storage does not eliminate the need for local disk entirely. Recent (hot) data still resides on broker disks for low-latency consumption. Broker local storage is still required for active segments.
Reading from the tiered (S3) tier has higher latency than reading from local disk. Consumer applications that replay old data will experience S3 GET latency.
Not all Kafka distributions implement KIP-405 identically. Confluent's implementation differs from Apache Kafka's in configuration and maturity.

Key Connections

scoped_to S3, Object Storage — offloads Kafka log segments to S3
enables Event-Driven Ingestion — long-retention event streams without broker scaling
used_by Debezium — CDC events benefit from extended retention on S3
relates_to Tiered Storage — Kafka-specific instance of the tiered storage pattern

Definition

What it is

A Kafka feature (KIP-405) that offloads older log segments from local broker disks to S3-compatible object storage, enabling virtually unlimited retention without scaling broker storage.

Why it exists

Kafka brokers traditionally store all retained data on local disk, forcing a tradeoff between retention period and disk cost. Tiered storage breaks this constraint by moving cold segments to S3, keeping only hot data on fast local storage.

Primary use cases

Long-term Kafka log retention on S3, cost-effective event replay from object storage, decoupling Kafka retention from broker disk capacity.

Recent developments

Latest signals

Apache Kafka 4.2.0 shipped (Feb 17, 2026), carrying the tiered-storage-adjacent diskless work. Alongside KIP-405 tiered storage, the 4.2 line introduces KIP-1267 (Tiered Storage Cost Attribution Metrics) and KIP-1183 (Unified Shared Storage), per InfoQ's cloud-native Kafka architecture write-up. The Red Hat digest confirms the release cadence continued with 4.0.2 and 4.1.2 point releases and a 4.3.0 feature freeze on March 18, 2026.
Diskless topics (KIP-1150) formally accepted March 2, 2026 — the successor to broker-local tiered storage. KIP-1150 establishes S3-as-primary "Diskless Topics" in Apache Kafka itself; KIP-1176 (Tiered Storage for Active Log Segment) was withdrawn in December 2025 in favor of it, per Instaclustr's Kafka Diskless status page. Follow-on KIP-1163 (Diskless Core) and KIP-1164 are in active design as of April 2026. KIP-405 tiered storage remains the current production standard while diskless matures.
Tiered-storage spec surface is expanding via new KIPs. Per the Conduktor KIP explorer: KIP-1272 extends tiered storage (KIP-405) to support compacted topics; KIP-1269 extends the DescribeLogDirs RPC to report remote (tiered) log size per partition; and the March 2026 Kafka Monthly Digest lists KIP-1303 (Deprioritize Tiered Storage Followers In Leader Election).
Apache Kafka 4.0 tiered-storage operations doc — canonical operator reference. The Apache Kafka 4.0 Tiered Storage operations guide (last modified March 20, 2026) formalizes the config surface: remote.log.storage.system.enable=true flips the broker into two-tier mode; RemoteStorageManager is the pluggable interface for the remote backend (S3, HDFS, custom) — Kafka ships no default, operators choose; RemoteLogMetadataManager defaults to a Kafka-internal-topic implementation with strongly consistent semantics.
Managed-service adoption widening. Instaclustr's March 2026 product update put Tiered Storage into public review for Apache Kafka on Google Cloud Platform, extending beyond earlier AWS/Confluent/Aiven RemoteStorageManager implementations. The competitive frame holds: WarpStream and Redpanda went S3-as-primary, while Kafka's tiered storage is S3-as-cold-tier.
Conduktor quantifies the tiering payoff beyond storage cost: 3–9x storage cost reduction, 15x faster partition reassignment, 13x faster cluster scaling. The operational-speed figures (reassignment, scaling) are the less-obvious benefit — tiered brokers carry far less local data to reshuffle when partitions move. Per Tiered Storage in Kafka (Conduktor).
KIP-405's own test data shows tiering narrows, not eliminates, the historical-read latency tax. Without tiered storage, a historical-read consumer pushes p99 produce latency from 21ms to 60ms; with tiered storage enabled the same scenario runs 25ms to 42ms, and producer throughput still drops roughly 43% under that load. Per Kafka Tiered Storage: Reads, Deletes & Data Flow (Aiven) and Apache Kafka Tiered Storage - KIP-405 (2minutestreaming).
"Move data to S3" isn't the whole cost story — six cost buckets stay in play. AutoMQ's cost-model breakdown counters simplistic tiering-savings narratives: broker disk, object storage capacity, object requests, network, replication/recovery, and operations all still factor in, and tiered storage doesn't zero out most of them. Worth weighing against vendor "3-9x savings" headline figures above. Per Kafka Tiered Storage Cost: Broker Disk, Object Storage, and Network Fees (AutoMQ).
Local retention is typically 10-20x smaller than the remote tier in production deployments. That ratio is the practical shape of "hot local, cold S3" — most of the data footprint sits in the remote tier, with only a thin recent-data slice on local broker disk. Per Apache Kafka Tiered Storage in Depth: How Writes and Metadata Flow (Aiven). Sources: Apache Kafka 4.0 Tiered Storage ops guide (kafka.apache.org) · InfoQ — Cloud-Native Kafka · Instaclustr — Kafka Diskless status · Conduktor KIP explorer · Kafka Monthly Digest March 2026 (developers.redhat.com) · Instaclustr product update March 2026