Technology

Kafka Tiered Storage

An Apache Kafka feature (KIP-405) that offloads older log segments from broker-local disks to S3-compatible object storage, extending Kafka's retention capacity without scaling broker storage proportionally.

10 connections 3 resources

Summary

What it is

Where it fits

Kafka Tiered Storage bridges the gap between real-time event streaming and long-term S3 storage. By transparently moving cold log segments to S3, it allows Kafka to serve as both the streaming platform and a long-retention event archive, reducing the need for separate S3 sink connectors for archival.

Misconceptions / Traps

Tiered storage does not eliminate the need for local disk entirely. Recent (hot) data still resides on broker disks for low-latency consumption. Broker local storage is still required for active segments.
Reading from the tiered (S3) tier has higher latency than reading from local disk. Consumer applications that replay old data will experience S3 GET latency.
Not all Kafka distributions implement KIP-405 identically. Confluent's implementation differs from Apache Kafka's in configuration and maturity.

Key Connections

scoped_to S3, Object Storage — offloads Kafka log segments to S3
enables Event-Driven Ingestion — long-retention event streams without broker scaling
used_by Debezium — CDC events benefit from extended retention on S3
relates_to Tiered Storage — Kafka-specific instance of the tiered storage pattern

Definition

What it is

A Kafka feature (KIP-405) that offloads older log segments from local broker disks to S3-compatible object storage, enabling virtually unlimited retention without scaling broker storage.

Why it exists

Kafka brokers traditionally store all retained data on local disk, forcing a tradeoff between retention period and disk cost. Tiered storage breaks this constraint by moving cold segments to S3, keeping only hot data on fast local storage.

Primary use cases

Long-term Kafka log retention on S3, cost-effective event replay from object storage, decoupling Kafka retention from broker disk capacity.

Recent developments

Latest signals

Apache Kafka 4.2.0 shipped (Feb 17, 2026), carrying the tiered-storage-adjacent diskless work. Alongside KIP-405 tiered storage, the 4.2 line introduces KIP-1267 (Tiered Storage Cost Attribution Metrics) and KIP-1183 (Unified Shared Storage), per InfoQ's cloud-native Kafka architecture write-up. The Red Hat digest confirms the release cadence continued with 4.0.2 and 4.1.2 point releases and a 4.3.0 feature freeze on March 18, 2026.
Diskless topics (KIP-1150) formally accepted March 2, 2026 — the successor to broker-local tiered storage. KIP-1150 establishes S3-as-primary "Diskless Topics" in Apache Kafka itself; KIP-1176 (Tiered Storage for Active Log Segment) was withdrawn in December 2025 in favor of it, per Instaclustr's Kafka Diskless status page. Follow-on KIP-1163 (Diskless Core) and KIP-1164 are in active design as of April 2026. KIP-405 tiered storage remains the current production standard while diskless matures.
Tiered-storage spec surface is expanding via new KIPs. Per the Conduktor KIP explorer: KIP-1272 extends tiered storage (KIP-405) to support compacted topics; KIP-1269 extends the DescribeLogDirs RPC to report remote (tiered) log size per partition; and the March 2026 Kafka Monthly Digest lists KIP-1303 (Deprioritize Tiered Storage Followers In Leader Election).
Apache Kafka 4.0 tiered-storage operations doc — canonical operator reference. The Apache Kafka 4.0 Tiered Storage operations guide (last modified March 20, 2026) formalizes the config surface: remote.log.storage.system.enable=true flips the broker into two-tier mode; RemoteStorageManager is the pluggable interface for the remote backend (S3, HDFS, custom) — Kafka ships no default, operators choose; RemoteLogMetadataManager defaults to a Kafka-internal-topic implementation with strongly consistent semantics.
Managed-service adoption widening. Instaclustr's March 2026 product update put Tiered Storage into public review for Apache Kafka on Google Cloud Platform, extending beyond earlier AWS/Confluent/Aiven RemoteStorageManager implementations. The competitive frame holds: WarpStream and Redpanda went S3-as-primary, while Kafka's tiered storage is S3-as-cold-tier.