Standard

Delta Lake Protocol

The specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based commit log stored alongside data files.

5 connections 4 resources

Summary

What it is

The specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based commit log stored alongside data files.

Where it fits

The Delta protocol is what makes Delta Lake tables transactional. The commit log serializes changes so concurrent readers and writers see consistent state — even on S3, where atomic rename is unavailable.

Misconceptions / Traps
  • The Delta protocol requires either atomic rename or an external coordination mechanism (DynamoDB, Azure ADLS). On S3, multi-cluster writes are unsafe without a log store.
  • Protocol versions (reader/writer features) must be managed carefully. Upgrading to a newer protocol version may make older readers unable to open the table.
Key Connections
  • enables Lakehouse Architecture — the spec that makes Delta Lake ACID possible
  • solves Schema Evolution — schema enforcement in the transaction log
  • Delta Lake depends_on Delta Lake Protocol
  • scoped_to Table Formats, Lakehouse

Definition

What it is

A specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based transaction log stored alongside data files.

Why it exists

To bring database-like reliability to data lakes. The Delta protocol ensures that concurrent readers and writers see consistent table state, even on eventually consistent storage, by serializing changes through a commit log.

Primary use cases

ACID-compliant data lake tables, streaming + batch unification on S3, audit-trail via transaction log history.

Connections 5

Outbound 4
scoped_to2
Table FormatsLakehouse
enables1
Lakehouse Architecture
solves1
Schema Evolution
Inbound 1
depends_on1
Delta Lake

Resources 4

SpecHigh
github.com/delta-io/delta/blob/master/PROTOCOL.md

The formal Delta Lake transaction log protocol specification defining the JSON action format, commit rules, checkpointing, schema enforcement, and time travel semantics.

DocsHigh
docs.delta.io/latest/index.html

Official Delta Lake documentation covering usage with Spark, API reference, table utilities, and operational guidance.

GitHubHigh
github.com/delta-io/delta

Canonical open-source repository for the Delta Lake project, containing the reference Spark-based implementation.

GitHubMedium
github.com/delta-incubator/delta-kernel-rs

Delta Kernel in Rust provides a standalone, engine-agnostic implementation of the Delta protocol, important for the multi-engine ecosystem.