Technology

DataFusion

An extensible query execution framework written in Rust, built on Apache Arrow, that provides a SQL query planner and execution engine for building custom analytics applications over S3-stored data.

7 connections 3 resources 1 post

Summary

What it is

An extensible query execution framework written in Rust, built on Apache Arrow, that provides a SQL query planner and execution engine for building custom analytics applications over S3-stored data.

Where it fits

DataFusion is the embedded query engine layer used by projects like Ballista, InfluxDB IOx, and Delta-rs. Rather than being a standalone analytics product, it is the foundation that other S3-native tools build upon for SQL query planning and columnar execution.

Misconceptions / Traps

DataFusion is a library, not a database. It provides query planning and execution but requires integration work to become a deployable analytics system.
DataFusion's Rust implementation offers memory safety and performance but limits extensibility to Rust or languages with Rust FFI bindings (Python via PyO3, C via extern).
Distributed execution requires Ballista or a custom scheduler. DataFusion alone runs single-node.

Key Connections

scoped_to S3, Lakehouse — query execution over S3-stored data
depends_on Apache Arrow — Arrow columnar format is the in-memory representation
depends_on Apache Parquet — reads Parquet files from S3
enables Apache Iceberg — used by the iceberg-rust implementation

Definition

What it is

An extensible, embeddable query engine written in Rust, built on Apache Arrow. Provides SQL and DataFrame APIs for querying data on S3, used as the query core in tools like Ballista, InfluxDB IOx, and Delta-rs.

Why it exists

Many projects need a fast, embeddable SQL engine that can read from S3 without deploying a full distributed query cluster. DataFusion provides a modular, Arrow-native query engine that can be embedded into Rust, Python, or other applications.

Primary use cases

Embedded SQL analytics over S3, building custom query engines on object storage, serverless query execution against Parquet/Iceberg on S3.

Recent developments

Latest signals

Latest release: v54.0.0 (GA June 12, 2026). Adds LATERAL joins, SQL lambda functions (array_transform, array_filter, array_any_match), content-defined chunking for Parquet, and a new Avro reader, plus join/scan/planning performance wins. ~740 commits from 139 contributors. Per DataFusion 54.0.0 release.
New subproject: Apache DataFusion Java 0.1.0 (May 26, 2026). First release of a thin Java binding over the DataFusion engine via a JNI bridge — extends DataFusion's native embedding beyond Rust/Python to the JVM. Per Introducing DataFusion Java 0.1.0.
Ballista 53.0.0 (May 24, 2026) — distributed query engine. The distributed execution engine built on DataFusion shipped 53.0.0, covering updates since the 43.0.0 line and enabling parallel workload execution across multiple nodes. Per Ballista 53.0.0 release.
Comet Spark accelerator now at 0.17.0 (June 20, 2026). The Comet subproject — which translates Spark physical plans to DataFusion physical plans without code changes — advanced from 0.15.0 to 0.17.0 in two months (0.16.0 May 7, 0.17.0 June 20), each release cutting fallbacks to Spark. A transparent perf path for shops that want DataFusion execution without rewriting Spark jobs. Per the DataFusion blog.
DataFusion 53.0.0 (April 2, 2026) — performance-driven release. DataFusion 53.0.0 shipped with 114 contributors and three structural wins: LIMIT-aware Parquet row group pruning, expanded filter pushdown through more join types and UnionExec plus dynamic-filter pushdown, and faster query planning via cheaper-to-clone immutable plan pieces.
Ecosystem footprint — the de facto Rust query engine. Per a GreptimeDB / DataFusion PMC retrospective, DataFusion now powers roughly 3,000 GitHub repositories as a library (Spice.ai, GreptimeDB, InfluxDB IOx, Delta-rs, Vortex, and others); the core crate sees ~1.09M downloads/month.