Technology

DuckLake

A lakehouse metadata format that stores table metadata in an embedded SQL database (DuckDB) instead of file-based manifests on S3. Emerging project from the DuckDB team.

11 connections 2 resources 1 post

Summary

What it is

A lakehouse metadata format that stores table metadata in an embedded SQL database (DuckDB) instead of file-based manifests on S3. Emerging project from the DuckDB team.

Where it fits

DuckLake challenges the Iceberg/Delta approach of storing metadata as JSON and Avro files on S3. By placing metadata in a SQL database, it eliminates the metadata file listing and parsing overhead that plagues large Iceberg tables — while keeping data files in Parquet on S3. It is the natural extension of DuckDB's "zero-infrastructure" philosophy to the lakehouse metadata layer.

Misconceptions / Traps

DuckLake is early-stage (2025). It is not a production-ready replacement for Iceberg or Delta Lake. Evaluate for experimentation and single-node workflows, not mission-critical multi-engine environments.
The metadata database (DuckDB, PostgreSQL, MySQL) becomes a stateful dependency. This partially trades the "no server needed" benefit of file-based table formats for a database dependency.
Multi-engine support is limited. DuckLake is tightly coupled to DuckDB today — unlike Iceberg, which works across Spark, Trino, Flink, and others.

Key Connections

depends_on DuckDB — uses DuckDB as the embedded metadata engine
alternative_to Apache Iceberg — SQL-based metadata vs file-based manifests
solves Metadata Overhead at Scale — eliminates file-based metadata listing overhead
solves Request Amplification — metadata queries replace S3 LIST and GET operations

Definition

What it is

An open MIT-licensed table format that stores all lakehouse metadata — schemas, transaction logs, file locations, version histories, snapshot isolations — inside an ACID-compliant SQL database (PostgreSQL, MySQL, embedded DuckDB, or SQLite) rather than as immutable JSON/Avro files on object storage. Physical data still lives as standard Parquet files on S3-compatible blob storage; only the control plane moves into the RDBMS. Authored by the DuckDB team (Mark Raasveldt and Hannes Mühleisen of CWI Amsterdam) and published as the "DuckLake Manifesto: SQL as a Lakehouse Format."

Why it exists

File-based table formats (Iceberg, Delta, Hudi) treat S3 as both the data store *and* the transaction log. That works for batch analytics but produces severe S3 request amplification under streaming or AI workloads: a single 10,000-row update against a 100M-row Iceberg table triggers a multi-step sequence of metadata.json read → manifest-list.avro download → manifest files traversal → new Parquet writes → position-delete files → new manifests → new manifest-list → atomic catalog swap, generating ~24+ S3 API calls per micro-batch. DuckLake collapses metadata I/O into a single indexed SQL query against an RDBMS, eliminating the "Manifest Maze" entirely.

Primary use cases

Low-latency lakehouse metadata operations, real-time CDC into the lakehouse, interactive embedded analytics where multi-second query planning is unacceptable, single-team to mid-market deployments where a managed PostgreSQL can handle the metadata throughput, AI agent retrieval pipelines that need sub-second context lookups.

Recent developments

Latest signals

DuckLake 1.0 GA (April 13, 2026) — production-ready, backward-compatibility guaranteed. The spec and the ducklake DuckDB extension reached 1.0, with the reference implementation shipping in DuckDB v1.5.2. 1.0 adds sorted tables, bucket partitioning, data inlining, geometry support, and Iceberg-compatible deletion vectors — so DuckLake's database-backed metadata now interoperates with the same deletion-vector encoding Iceberg V3 uses. The progression v0.3 → v0.4 (DuckDB 1.5.0, March) → v1.0 took the format from manifesto to production in under a year; the v1.1 standard is targeted for September 2026. Per DuckLake v1.0 Reaches Production-Readiness and Announcing DuckLake 1.0 on MotherDuck.
1.0 feature depth: VARIANT, GEOMETRY, Frozen DuckLakes — and a 105×/926× benchmark vs Iceberg. The 1.0 spec adds a binary VARIANT type with shredded-JSON pushdown, GEOMETRY spatial pushdown via bounding-box stats, and Frozen DuckLakes — read-only lakehouses served straight from public HTTPS with no auth or compute layer. v0.3's bidirectional metadata-only Iceberg copy (snapshot history preserved, no Parquet rewrite) makes DuckLake↔Iceberg a translation, not a migration. MotherDuck's mid-2026 benchmark credits the SQL-catalog metadata path with up to 105× faster streaming ingestion and 926× faster aggregation vs Iceberg. Per DuckLake 1.0, DuckLake 1.0 on MotherDuck, and DuckLake as a lakehouse format (Sparking Scala).
The clearest instance of "the catalog becoming a database." DuckLake's whole thesis — store all lakehouse metadata in an RDBMS (PostgreSQL/SQLite/DuckDB) instead of scattered files — is the production embodiment of the shift covered in The Catalog Is Becoming a Database. The Register framed the 1.0 release specifically around using an RDBMS to solve the lakehouse "small-changes" problem. Per DuckDB uses RDBMS to tackle lakehouse 'small changes' issue (The Register).
The DuckLake Manifesto landed as the spec foundation. Per the DuckLake Manifesto, DuckLake is positioned as "SQL as a Lakehouse Format" — an explicit philosophical break from the file-based-metadata orthodoxy of Iceberg/Delta/Hudi. The format is open MIT, not proprietary to DuckDB; standard tooling at ducklake.select and endjin's hands-on tutorial show production setups against PostgreSQL or MySQL backends. The architectural framing has shifted from "another open table format" to "the third generation of lakehouse architecture — database-backed metadata."
DuckDB 1.4 LTS — production readiness signal (October 2025). Per the DuckDB business-case analysis, DuckDB 1.4 LTS (October 2025) shipped AES-256 encryption, native Iceberg writes, and a rewritten sort engine — definitively signaling enterprise readiness for the underlying engine that powers DuckLake's embedded variant. Stack Overflow Developer Survey adoption jumped from 1.4% → 3.3% in one year; the project surpassed 30,000 GitHub stars; ClickBench gives DuckDB the #1 position for in-memory analytical workloads.
The Duck Stack economic case — 70% TCO reduction. Definite, an analytics platform provider, migrated their entire production infrastructure from Snowflake to the "Duck Stack" (DuckDB + DuckLake) and reports 70% reduction in underlying infrastructure costs — see DuckDB and DuckLake: Why We Bet the Company. Concrete numbers: standard object storage at ~$20/TB/month, plus a single dedicated 16-vCPU/64GB VM at ~$500/month with zero per-query execution charges, enabled sub-second query latency at a fraction of cloud-warehouse TCO. The shift lets Definite offer an all-in-one analytics platform from $250/month.
Operator's deployment matrix. Per Duck Lake vs Iceberg: An Operator's Verdict, the production sweet spot is workloads under 5 TB — micro-scale (≤100 GB) maps to local DuckDB/SQLite catalogs; mid-market (100 GB – 5 TB) maps to managed PostgreSQL catalogs; enterprise core (1–50 TB) is hybrid / engine-dependent; large-enterprise and hyperscale (>5 PB) still favor Iceberg's decentralized optimistic-concurrency model for multi-engine federation. Critically, migration between formats is reversible — both use standard Parquet for physical data, so DuckLake↔Iceberg conversions are metadata-translation operations, not physical-data rewrites. Architectural decisions made today are not permanent traps.
The AI-retrieval latency case (the strongest 2026 argument). Per the Designing an AI-Native Lakehouse on Iceberg engineering case study, AI agents querying an Iceberg-hosted vector embeddings table see median end-to-end retrieval latencies around 680ms with p95 at 1.8s, and 40–60% of that time is exclusively manifest/snapshot lookups on object storage — independent of the actual vector similarity search. DuckLake's single-SQL-call metadata path eliminates that entire latency component, returning the file URI list in milliseconds.
Tigris's hosted DuckLake offering. Per Tigris's DuckLake writeup, the geo-distributed S3-compatible object store now bundles DuckLake support as a managed pattern — single global namespace, edge replication of the catalog, zero-egress economics. Useful when the deployment requirement crosses regulatory boundaries that AWS S3's residency model can't satisfy.
MotherDuck — hybrid DuckLake-cloud. Per Quacks & Stacks, MotherDuck (the hosted DuckDB platform by the DuckDB team) is positioning as the "glue" that lets a single SQL query join local high-velocity DuckLake metadata against historical petabyte-scale Iceberg data on S3 — without the analyst or AI agent needing to know where the physical network boundary lies.
The scalability ceiling — honest about where it doesn't fit. Per the Rethinking the Lakehouse operator analysis: the centralized PostgreSQL catalog is a write-coordination bottleneck above ~50 TB / thousands of concurrent distributed writers. For hyperscale globally-distributed write fan-out, Iceberg's decentralized optimistic concurrency model remains the correct architectural choice despite its S3 amplification flaws. DuckLake is not a universal Iceberg replacement; it is the correct format for the workload tier where the centralized catalog's vertical simplicity beats Iceberg's horizontal complexity.
Where Iceberg is converging (long-term). Per the data-lakehouse hub 2025/2026 guide, the Iceberg V4 spec direction is moving toward a pluggable catalog model with RDBMS-backed metadata as an official option — effectively absorbing DuckLake's core thesis while maintaining Iceberg's massive ecosystem footprint. By the late 2020s the ideological divide between file-based and database-backed metadata is expected to narrow significantly. DuckLake's bet is that being early to the SQL-native pattern earns the format the developer-experience defaults for the embedded-and-edge tier even as Iceberg catches up at the hyperscale tier.
MotherDuck's April 2026 ecosystem readout puts a fresh number on the SQL-catalog thesis: metadata-only COUNT(*) queries run 8x–258x faster than the equivalent Iceberg scan, drawn from 108 PRs merged since late 2025 (68 of them reliability-focused). Per DuckDB Ecosystem Newsletter: April 2026 (MotherDuck).
pg_ducklake, the Postgres-native DuckLake catalog implementation, cracked ClickBench's top 10 and inserts small batches 5.8x faster than the DuckDB+DuckLake combination, per the project's own data-inlining benchmark. Tertiary, vendor-sourced. Per Data Inlining in pg_ducklake and PostgreSQL is now ClickBench Top-10.
pg_duckpipe closes the ingestion side: WAL-based CDC from Postgres heap tables straight into DuckLake, no Kafka. A companion extension streams PostgreSQL changes into DuckLake columnar tables on S3 via logical-decoding CDC, with per-table state machines managing snapshot, catchup, and streaming phases independently — the transactional-to-analytical pipeline collapses into two Postgres extensions and a bucket. Per pg_duckpipe: Real-time CDC for streaming Postgres tables into columnar DuckLake (dev.to).
MotherDuck's own materials describe DuckLake as built for "petabyte-scale lakehouses," with metadata operations 10-100x faster than Iceberg/Delta and support for millions of snapshots — a rosier framing than the roughly 50TB centralized-catalog ceiling independent operator analysis has flagged for the same architecture. Per Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data (MotherDuck). Sources: DuckLake Manifesto (ducklake.select) · DuckLake home (ducklake.select) · DuckLake & Iceberg: Modern Lakehouse Architecture 2025 (sanj.dev) · Getting Started with DuckLake (MotherDuck) · Quacks & Stacks: DuckLake's one-table wonder (Medium / tfmv) · DuckLake in Practice: Hands-On Tutorial (endjin) · DuckDB and DuckLake: Why We Bet the Company (Definite) · Duck Lake vs Iceberg: An Operator's Verdict (Definite) · Rethinking the Lakehouse (Medium) · Get your data ducks in a row with DuckLake (Tigris) · Designing an AI-Native Lakehouse on Iceberg (Medium) · 2025/2026 Guide to Data Lakehouses · Ultimate Guide to Open Table Formats — Iceberg, Delta, Hudi, Paimon, DuckLake (DEV)