DuckDB
An in-process analytical database engine (like SQLite for analytics) that reads Parquet, Iceberg, and other formats directly from S3 without requiring a server or cluster.
Summary
An in-process analytical database engine (like SQLite for analytics) that reads Parquet, Iceberg, and other formats directly from S3 without requiring a server or cluster.
DuckDB fills the gap between "I need to explore this S3 data" and "I need to deploy a Spark cluster." It brings fast columnar analytics to a single machine, reading S3 data directly — ideal for development, ad-hoc analysis, and embedded analytics.
- DuckDB is single-node. It does not scale horizontally. For petabyte-scale queries, you still need Spark, Trino, or StarRocks.
- DuckDB reads from S3 over HTTP. Performance is bottlenecked by network throughput and S3 request latency, especially with many small files.
depends_onApache Parquet, Apache Arrow — reads Parquet, processes in Arrow formatconstrained_bySmall Files Problem, Object Listing Performance — performance degrades with too many small S3 objects- Natural Language Querying
augmentsDuckDB — LLMs can generate SQL for DuckDB scoped_toS3, Lakehouse
Definition
An in-process analytical database engine (similar to SQLite for analytics) that can directly read Parquet, Iceberg, and other formats from S3 without requiring a server or cluster.
Not every analytical query requires a distributed cluster. DuckDB brings fast columnar analytics to a single machine, reading directly from S3 — eliminating the need to copy data to a local database or set up distributed infrastructure.
Local S3 data exploration, ad-hoc analytics over Parquet files on S3, development and testing of queries before deploying to distributed engines, embedded analytics.
Recent developments
- DuckDB 1.5.0 ("Variegata") — major feature release. The headline is a redesigned friendly CLI with significantly better ergonomics for ad-hoc data exploration (better column truncation, prompt cues, history). Two new first-class types landed: VARIANT for semi-structured / JSON-shaped data (so you no longer round-trip through string columns when ingesting heterogeneous payloads), and a built-in GEOMETRY type for spatial workloads (replacing the prior need for a third-party geo extension). Improved Iceberg integration — DuckDB can now read Iceberg V3 tables more reliably, including the deletion-vector encoding that landed in Iceberg V3.
- Point releases shipped on a tight cadence. 1.5.1 rolled bug fixes plus a CRAN-published R package (the project is taking the R ecosystem seriously now, not just Python). 1.5.2 (April 2026) was a performance-focused release with hot-path improvements in the columnar scan engine.
- Release calendar — what's coming. 1.5.3 is tracked for May 18, 2026 (likely another stability point release before 2.0). 2.0.0 is on the calendar for September 2026 — given the project's track record, expect a major overhaul of the storage format and possibly the query plan optimizer. Starting with 1.4.0, the project formalized an LTS rhythm: every other minor version is designated long-term-support, so operators can pick a stable line without giving up ongoing fixes.
- Native vector similarity search arrived 2025-2026. DuckDB now has built-in HNSW-style vector search, putting it in direct competition with PostgreSQL + pgvector for embedded RAG use cases. The pitch: instead of standing up a separate Postgres instance just to host pgvector, you can run DuckDB in-process inside your application, query the vector index alongside your tabular data, and skip the network hop entirely. For applications already using DuckDB for analytics, the vector layer is essentially free.
- Positioning shift — DuckDB is becoming the Pandas replacement. Industry framing in 2026 has decisively shifted: DuckDB is no longer "the embedded analytical query engine" — it's "the default tool for large-scale single-machine analysis," displacing Pandas in places where data exceeded RAM or required SQL-shaped operations. Combined with the Iceberg/Parquet integration on the read side and the new VARIANT type on the ingest side, DuckDB sits in an unusually strong position: capable of handling ~100 GB single-node workloads end-to-end without a distributed cluster.
Connections 18
Outbound 8
depends_on2constrained_by2alternative_to2Inbound 10
augments2alternative_to2Resources 3
Official DuckDB documentation covering SQL dialect, extensions, and embedded analytics engine capabilities.
Primary DuckDB repository with the full C++ source, extension framework, and build system.
DuckDB's dedicated S3 support documentation covering direct S3 reads/writes via the httpfs extension, credentials configuration, and Parquet-on-S3 queries.