pg_duckpipe
A PostgreSQL extension streaming heap tables into DuckLake columnar tables on S3 via WAL-based CDC — the transactional-to-analytical pipeline reduced to one Postgres extension, no Kafka or Debezium.
Summary
A PostgreSQL extension streaming heap tables into DuckLake columnar tables on S3 via WAL-based CDC — the transactional-to-analytical pipeline reduced to one Postgres extension, no Kafka or Debezium.
The ingestion companion to pg_ducklake in the "Duck Stack": pg_ducklake gives Postgres a DuckLake catalog, pg_duckpipe keeps it fed from live tables. Per-table state machines (snapshot → catchup → streaming) isolate lag to individual tables.
- Author-documented only so far — no independent production reports; treat throughput expectations accordingly.
- Single-Postgres-instance scope; it is not a distributed CDC platform like Flink CDC.
depends_onDuckLake — the catalog + storage layer it writes intoalternative_toDebezium / Supermetal — same CDC job, embedded in Postgres itself
Definition
A PostgreSQL extension that streams heap tables into [DuckLake](/node/ducklake) columnar tables on S3 in real time, using Write-Ahead Log (WAL) based change data capture — no Kafka, no Debezium, no external broker infrastructure.
The classic transactional-to-analytical pipeline (Postgres → Debezium → Kafka → sink connector → lakehouse) carries three systems' worth of operational burden for what is conceptually one data movement. pg_duckpipe collapses it into a single Postgres extension: logical decoding feeds DuckLake's SQL-catalog metadata path directly, so changed rows land as columnar Parquet on object storage with lakehouse snapshot semantics.
Single-node lakehouse ingestion, real-time analytics over operational Postgres data without a streaming platform, incremental Parquet materialization for AI retrieval corpora, low-footprint CDC in edge and self-hosted deployments.
Connections 4
Outbound 4
Resources 2
The author's architectural writeup — WAL-based CDC into DuckLake, per-table snapshot/catchup/streaming state machines, no broker tier. Currently the primary available documentation.
The companion pg_ducklake catalog extension this pipeline feeds into — together they form the Postgres-native DuckLake stack.