Getting Data Into the Lakehouse — Choosing a CDC-to-Iceberg Path in 2026

The default way to move operational data into the lakehouse used to be a four-system pipeline: Debezium reads the database WAL, Kafka buffers it, Flink transforms it, Parquet lands on S3. In 2026 that architecture is optional at every hop — single-process Rust tools snapshot Postgres into Iceberg in 13 minutes where the pipeline took 90+, streaming SQL databases replace the pipeline and the serving layer, and Python-native processors run at 25× lower memory than the JVM incumbents. The question is no longer "how do we operate the pipeline" but "which of these five shapes fits our workload." This guide is the decision path.

• Topics: S3, Data Lake, Table Formats
• Technologies: Flink CDC, Supermetal, RisingWave, Debezium, Bytewax, Estuary Flow, Apache Flink, Apache Spark, Apache Iceberg, Apache Paimon
• Architectures: CDC into Lakehouse, Lakehouse Architecture
• Pain Points: Legacy Ingestion Bottlenecks

1. First decide: does your transformation logic fit SQL? This forks the whole tree.

   • Yes, and you also need a serving layer (dashboards, alerts, sub-second materialized views) → RisingWave. One Postgres-compatible system replaces pipeline + OLAP store + glue. No Kafka, no JVM.
   • Yes, but you just need rows landed in Iceberg → keep reading; you're choosing an ingestion tool, not a streaming database.
   • No — you need code-level operators → Flink CDC (JVM, distributed) or Bytewax (Python, single-node). See step 4.

2. Sizing the snapshot matters more than sizing the stream. Initial snapshots are where pipelines die. On the normalized single-node benchmark (TPC-H SF50, ~375M rows): Supermetal 13 min, Flink CDC 90–116 min (tuned), Kafka Connect/Debezium 120 min, Spark 200+ min. If your databases are large and your team is small, the single-process Rust path (WAL → Arrow → Parquet, phase-aware Iceberg writer) is the default choice, not the exotic one.

3. Pick Flink CDC when you genuinely need its ceiling. Distributed exactly-once across a multi-node cluster, sub-second end-to-end latency at high sustained throughput, schema evolution that propagates ALTER TABLE into Iceberg automatically (3.6.0 added PostgreSQL + the VARIANT type for semi-structured payloads), and the broadest source/sink matrix (Oracle source, Hudi sink). The price: JVM cluster operations — checkpoints, RocksDB state backends, JobManager/TaskManager care. Pay it when the workload justifies it, not by default.

4. Keep Debezium where Kafka is the product, not the plumbing. If multiple downstream consumers genuinely fan out from the change stream (search indexers, caches, microservices and the lakehouse), Debezium + Kafka Connect remains the right shape — that's what it's for. If the lakehouse is the only consumer, the Kafka hop is pure overhead; use a direct path.

5. Python-native teams: Bytewax, with eyes open. 25× lower memory than Flink and 4.6× lower TCO on the McKnight benchmark — but the project is community-maintained since May 2025 with a stalled release cadence. Right for moderate-throughput embedding/feature pipelines owned by ML engineers; wrong for the mission-critical core path.

6. Managed option: Estuary Flow. Sub-100ms latency, exactly-once, 200+ connectors, GB-based pricing — the "we don't want to operate any of this" answer, with the usual managed-service data-residency caveats (BYOC deployment mitigates).

7. Whatever you pick, land in Iceberg. Every tool in this guide writes Iceberg natively; the format is the read-path interchange layer (see Guide 48 for who governs it). Landing anywhere else in 2026 re-creates the copy problem this architecture exists to solve.