Technology

Debezium

An open-source distributed platform for change data capture (CDC) that streams row-level changes from databases (PostgreSQL, MySQL, MongoDB, and others) into event streams, enabling real-time ingestion into S3-based lakehouses.

9 connections3 resources2 posts

Summary

What it is

An open-source distributed platform for change data capture (CDC) that streams row-level changes from databases (PostgreSQL, MySQL, MongoDB, and others) into event streams, enabling real-time ingestion into S3-based lakehouses.

Where it fits

Debezium sits at the ingestion boundary between operational databases and the S3 data lake. It captures INSERT, UPDATE, and DELETE events from database transaction logs and publishes them to Kafka, from which downstream connectors write to S3 in Parquet or Iceberg format.

Misconceptions / Traps
  • Debezium captures changes but does not write directly to S3. It requires a downstream sink (Kafka Connect S3 Sink, Flink, or a table format writer) to land data on object storage.
  • CDC from databases generates many small events. Without batching and compaction downstream, this creates the small files problem on S3.
  • Schema changes in the source database propagate through Debezium as schema change events. If the lakehouse layer does not handle schema evolution, pipeline breakage occurs.
Key Connections
  • scoped_to S3, Lakehouse — CDC ingestion into S3-based lakehouses
  • enables CDC into Lakehouse — the primary architecture pattern Debezium feeds
  • used_by Apache Flink, Apache Spark — stream processors that consume Debezium events
  • depends_on Kafka Tiered Storage, Redpanda — message brokers that transport CDC events

Definition

What it is

An open-source distributed platform for change data capture (CDC) that streams row-level changes from databases (PostgreSQL, MySQL, MongoDB, and others) into downstream systems such as Kafka, which can then land data into S3-based lakehouses.

Why it exists

Getting data from transactional databases into S3-based data lakes traditionally requires batch ETL with full table scans. Debezium captures changes as they happen, enabling near-real-time ingestion into lakehouse tables without impacting source database performance.

Primary use cases

Real-time database replication to S3 lakehouses, CDC-driven Iceberg/Delta/Hudi ingestion, event-sourced data pipelines.

Recent developments

Latest signals
  • Latest release: 3.6.1.Final (August 2026). Succeeds 3.6.0.Final (early July 2026), which itself succeeded 3.5.2.Final; the parallel Quarkus extensions track the same versions. Per debezium/debezium tags.

  • Parallel multithreaded chunked snapshots landed in the 3.5 line. The incremental/initial snapshotter can now run chunk-based snapshots across multiple threads, directly targeting the long-standing single-threaded snapshot bottleneck. Per Debezium blog.

  • Debezium Server benchmarked for Kafka-less deployments. An official MySQL-to-Kafka benchmark shows default configs deliver stable near-real-time CDC, with source-database write capacity — not Debezium Server — as the throughput limiter, positioning Debezium Server for lightweight projects that want CDC without the full Kafka Connect stack. Per Measuring Debezium Server performance (Feb 2, 2026).

  • GSoC 2026: new-connector and AI work. Debezium joined Google Summer of Code 2026 with projects for a Milvus connector, a SQLite source connector, a Debezium CLI, and PyDebeziumAI. Per GSoC 2026 announcement.

  • Roadmap: 3.8 targeted Oct–Dec 2026. Planned additions include an ElasticSearch sink, a SQLite source connector, and IAM authentication; longer-term items include InfluxDB and Fluss sinks, a WASM sink, and multi-tenant SPI. Per Debezium roadmap.

  • Position vs Flink CDC holds: Debezium is the Kafka-Connect-anchored path; Flink CDC is the no-Kafka path. Debezium remains the most widely deployed CDC tool for Kafka fan-out, while Flink CDC suits teams writing directly to Iceberg/Paimon/Hudi without the Kafka hop. Per RisingWave's Debezium alternatives survey and Conduktor's CDC guide.

  • Incubating connectors in active development for Vitess, Spanner, Informix, and IBM i, alongside the core connector, website, schema registry, Quarkus extensions, and Kubernetes operator repos. Per debezium/debezium organization repositories.

  • A real deployment sizing example: DataHub's own Debezium-based CDC path runs 50–200ms DB-write-to-metadata-emission latency at roughly 512MB–1GB RAM per Kafka Connect connector. Useful as a concrete resource-planning data point beyond Debezium's own benchmarks. Per DataHub — Configuring CDC Mode. (tertiary)

  • The streaming phase (not just the initial snapshot) remains a single-threaded bottleneck per connector — independent of the 3.5 parallel-snapshot improvement — capping Postgres connector throughput at roughly 7,000 events/sec, with at-least-once delivery (so downstream dedup is required) and manual intervention still needed for complex DDL schema-evolution cases. Per Debezium for CDC in Production: Pain Points and Limitations. (tertiary — vendor blog, but describes a specific architectural constraint rather than a comparative benchmark claim) Sources: debezium/debezium tags · Debezium Server performance (Feb 2026) · GSoC 2026 · Debezium roadmap · RisingWave alternatives survey

Connections9

Outbound6
Inbound3
alternative_to2

Resources3

Featured in