DuckDB
An in-process analytical database engine (like SQLite for analytics) that reads Parquet, Iceberg, and other formats directly from S3 without requiring a server or cluster.
Summary
An in-process analytical database engine (like SQLite for analytics) that reads Parquet, Iceberg, and other formats directly from S3 without requiring a server or cluster.
DuckDB fills the gap between "I need to explore this S3 data" and "I need to deploy a Spark cluster." It brings fast columnar analytics to a single machine, reading S3 data directly — ideal for development, ad-hoc analysis, and embedded analytics.
- DuckDB is single-node. It does not scale horizontally. For petabyte-scale queries, you still need Spark, Trino, or StarRocks.
- DuckDB reads from S3 over HTTP. Performance is bottlenecked by network throughput and S3 request latency, especially with many small files.
depends_onApache Parquet, Apache Arrow — reads Parquet, processes in Arrow formatconstrained_bySmall Files Problem, Object Listing Performance — performance degrades with too many small S3 objects- Natural Language Querying
augmentsDuckDB — LLMs can generate SQL for DuckDB scoped_toS3, Lakehouse
Definition
An in-process analytical database engine (similar to SQLite for analytics) that can directly read Parquet, Iceberg, and other formats from S3 without requiring a server or cluster.
Not every analytical query requires a distributed cluster. DuckDB brings fast columnar analytics to a single machine, reading directly from S3 — eliminating the need to copy data to a local database or set up distributed infrastructure.
Local S3 data exploration, ad-hoc analytics over Parquet files on S3, development and testing of queries before deploying to distributed engines, embedded analytics.
Recent developments
DuckDB 1.5.3 shipped May 20, 2026 — "not an ordinary patch release." It introduced the Quack client-server protocol as a core extension (DuckDB can now serve clients over a wire protocol, including Quack-in-DuckLake), alongside major DuckDB-Iceberg gains:
ALTER TABLEis now supported against Iceberg tables for the common schema-evolution operations, and DuckDB-Iceberg can create, insert into, and update tables using bucket and truncate partition transforms. Per DuckDB 1.5.3: Not an Ordinary Patch Release and New DuckDB-Iceberg Features in v1.5.3.1.5.1 added read/write support for the Lance lakehouse format via the
lancecore extension — extending DuckDB's multi-format reach beyond Parquet/Iceberg to the AI-native columnar format. Per Announcing DuckDB 1.5.1.DuckDB 1.5.0 ("Variegata") — major feature release. The headline is a redesigned friendly CLI with significantly better ergonomics for ad-hoc data exploration (better column truncation, prompt cues, history). Two new first-class types landed: VARIANT for semi-structured / JSON-shaped data (so you no longer round-trip through string columns when ingesting heterogeneous payloads), and a built-in GEOMETRY type for spatial workloads (replacing the prior need for a third-party geo extension). Improved Iceberg integration — DuckDB can now read Iceberg V3 tables more reliably, including the deletion-vector encoding that landed in Iceberg V3.
Point releases shipped on a tight cadence. 1.5.1 rolled bug fixes plus a CRAN-published R package (the project is taking the R ecosystem seriously now, not just Python). 1.5.2 (April 2026) was a performance-focused release with hot-path improvements in the columnar scan engine.
DuckDB 2.0 preview is official (August 17, 2026) — engine-wide asynchronous I/O, VARIANT as a first-class type, triggers, a new SQL parser, and a new storage format. The project published its own preview of the 2.0 release: the entire I/O layer becomes asynchronous (Parquet, CSV, and native format reads all scale independently of query processing — the change that matters most on network-attached object storage), with new
MMAPandDIRECT_IOmodes alongside it. VARIANT graduates to a first-class citizen for shredded-JSON pushdown, triggers and nested schemas push DuckDB into operational-database territory, and the Quack client-server protocol introduced in 1.5.3 stabilizes as a supported way to serve remote clients. Two deprecations to plan around before upgrading: the older lambda syntax and the Substrait API are being removed in favor of Pythonic lambdas. Per A Preview of DuckDB v2.0 (duckdb.org). The release itself is tracked for Fall 2026 on the official release calendar; starting with 1.4.0 the project formalized an LTS rhythm — every other minor version is designated long-term-support, so operators can pick a stable line without giving up ongoing fixes.Native vector similarity search arrived 2025-2026. DuckDB now has built-in HNSW-style vector search, putting it in direct competition with PostgreSQL + pgvector for embedded RAG use cases. The pitch: instead of standing up a separate Postgres instance just to host pgvector, you can run DuckDB in-process inside your application, query the vector index alongside your tabular data, and skip the network hop entirely. For applications already using DuckDB for analytics, the vector layer is essentially free.
Positioning shift — DuckDB is becoming the Pandas replacement. Industry framing in 2026 has decisively shifted: DuckDB is no longer "the embedded analytical query engine" — it's "the default tool for large-scale single-machine analysis," displacing Pandas in places where data exceeded RAM or required SQL-shaped operations. Combined with the Iceberg/Parquet integration on the read side and the new VARIANT type on the ingest side, DuckDB sits in an unusually strong position: capable of handling ~100 GB single-node workloads end-to-end without a distributed cluster.
pg_duckdb embeds DuckDB's vectorized OLAP engine directly inside a PostgreSQL server process, enabling analytical queries up to 1000x faster than native PostgreSQL in some benchmarks — adding DuckDB-speed analytics without standing up a separate database. Per Top 10 data warehouse platforms for 2026 (MotherDuck).
DuckDB now ships as a managed analytical read-only instance inside Alibaba Cloud's ApsaraDB RDS for MySQL, supporting HTAP query routing so analytical queries can be offloaded from the transactional primary without a separate pipeline. Per Expanded scope for DuckDB analytical read-only instances (Alibaba Cloud).
DuckDB 1.5.4 shipped June 22, 2026 with bug fixes, security hardening, and performance improvements — the point release immediately after 1.5.3's Quack protocol launch. Tertiary source (aggregator digest, no primary DuckDB blog post found in this pull). Per Data + AI 2026 Review, DuckDB 1.5.4 (tldr.tech).
DuckDB 2.0's fine print: a mandatory storage-format break, a stable Quack server protocol, and a new parser. The v2.0.0 file format (buffer-managed ART indexes, compact deletes, lazily loaded column metadata, DICT_FSST string compression by default) is backward-incompatible — older binaries cannot read it, and the community is pressing for zero-downtime migration guidance that doesn't yet exist. The
quackextension graduates to stable, letting an embedded session serve its tables over the network (CALL quack_serve()/ATTACH 'quack:host') — DuckDB's first sanctioned exit from single-process life. Legacyx -> x + 1lambda syntax is removed, and a bespoke PEG-based parser replaces the PostgreSQL-derived one. Whether 2.0.0 lands on an LTS cycle (every other minor release since 1.4 gets a year of community support) is the open enterprise question. Per dev.to — DuckDB v2.0 server mode, DuckDB release calendar, and DuckDB community support policy. Sources: DuckDB 2026 — In-Process Analytics Reshaping Data Science (programming-helper.com) · DuckDB 1.5.3 release (duckdb.org) · DuckDB-Iceberg v1.5.3 features
Connections 19
Outbound 9
depends_on2constrained_by2reads_from1alternative_to2Inbound 10
augments2alternative_to2Resources 3
Official DuckDB documentation covering SQL dialect, extensions, and embedded analytics engine capabilities.
Primary DuckDB repository with the full C++ source, extension framework, and build system.
DuckDB's dedicated S3 support documentation covering direct S3 reads/writes via the httpfs extension, credentials configuration, and Parquet-on-S3 queries.