Apache Hudi Spec
The specification for managing incremental data processing on object storage — record-level upserts, deletes, change logs, and timeline-based metadata.
Summary
The specification for managing incremental data processing on object storage — record-level upserts, deletes, change logs, and timeline-based metadata.
The Hudi spec defines how to efficiently mutate individual records in S3-stored datasets. It is the specification behind Hudi's Copy-on-Write and Merge-on-Read table types, and its timeline abstraction tracks all changes.
- The Hudi spec's timeline model is conceptually different from Iceberg's snapshot model and Delta's transaction log. Understanding the timeline abstraction is prerequisite to operating Hudi tables.
- The RFC-based evolution model means the spec is a living document. Breaking changes can be introduced via RFCs.
enablesLakehouse Architecture — makes incremental processing possible on data lakes- Apache Hudi
depends_onApache Hudi Spec scoped_toTable Formats, Lakehouse
Definition
The table-format specification underlying [Apache Hudi](/node/apache-hudi) — log-structured storage on object stores with first-class **record-level upserts**, **multi-table transactions** (Hudi 1.0 GA Jan 2025), **non-blocking concurrency control** for concurrent streaming writers, and **pluggable secondary indexing** (Bloom, R-tree, bitmap) sitting in the cloud metadata table. As of 2026, Hudi 1.0+ adds expression-based indexing (PostgreSQL-style) replacing traditional partitioning, partial updates (specific-field updates instead of whole-row rewrites), and a **pluggable table-format framework** that lets Hudi's transaction/index/concurrency-control machinery run on top of Apache Iceberg via [Apache XTable](/node/apache-xtable).
Pre-Hudi, "table format on S3" effectively meant "append-only log on S3 + snapshots" — Iceberg V2 / Delta — which optimizes for analytical-style writes but punishes record-level UPSERTs and CDC. Hudi inverts that: upserts are first-class, designed in from day one with copy-on-write + merge-on-read variants, and the multi-modal pluggable indexing subsystem makes record-level lookup performant at scale. The 2026 pluggable-table-format direction is the latest twist — letting Hudi's transaction layer run on top of *Iceberg* file layouts via XTable, hedging the format-vs-engine boundary.
Record-level upsert workloads where Hudi's MoR/CoW design dominates Iceberg's positional-delete approach, CDC pipelines into a lakehouse with sub-second commit latency, multi-table transactions for ACID across related Hudi tables (1.0 feature), AI/ML workloads with concurrent agent writes via non-blocking concurrency control, and the upcoming AI-shaped indexing patterns (vector embeddings as Bloom-indexable columns in Hudi 1.x).
Recent developments
- Hudi 1.0 GA — Jan 2025 — secondary indexing + multi-table transactions + non-blocking concurrency. General availability with SQL-based secondary indexes on arbitrary columns, multi-table transactions for cross-table ACID, and non-blocking concurrency control for concurrent streaming writers. Per InfoQ — Apache Hudi 1.0 GA.
- Pluggable table-format framework — Hudi storage engine over Iceberg via XTable. The 2026 headline feature: Hudi's transaction/index/concurrency-control engine becomes pluggable over the storage layer, letting it run on top of Iceberg file layouts through Apache XTable (Incubating). Per Apache Hudi 2025 Year in Review.
- AI/ML focused 2026 capabilities: unstructured types + column groups + vector search + Lance support. Upcoming Hudi releases planned for 2026 add unstructured data types, column groups optimized for embeddings, native vector search, and Apache Lance / Vortex format support — positioning Hudi as a lakehouse format for AI workloads. Per Apache Hudi 2025 Year in Review.
- Expression-based indexing replaces partitioning. A PostgreSQL-inspired feature in 1.0+: instead of partitioning physical data by a column, define SQL expressions over columns to drive query routing — more flexible than traditional partition keys. Per Onehouse — Hudi 1.0 Preview.
- Partial updates — field-level instead of row-level. Improves storage and compute efficiency by letting writers update specific fields of an existing row without rewriting the entire row. Per Medium — Apache Hudi 2026 guide.
Connections 5
Outbound 4
Inbound 1
depends_on1Resources 4
Technical specification pages documenting Hudi table format internals including the timeline, file layout, indexing mechanisms, and compaction strategies.
Official documentation covering table types (CoW and MoR), write operations, querying, and configuration.
Canonical repository containing the reference Java implementation, RFC documents, and the source of truth for the Hudi table format.
The Hudi RFC directory contains formal design documents for major features and format changes, serving as living specification amendments.