Standard

Delta Lake Protocol

The specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based commit log stored alongside data files.

6 connections 4 resources

Summary

What it is

The specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based commit log stored alongside data files.

Where it fits

The Delta protocol is what makes Delta Lake tables transactional. The commit log serializes changes so concurrent readers and writers see consistent state — even on S3, where atomic rename is unavailable.

Misconceptions / Traps

The Delta protocol requires either atomic rename or an external coordination mechanism (DynamoDB, Azure ADLS). On S3, multi-cluster writes are unsafe without a log store.
Protocol versions (reader/writer features) must be managed carefully. Upgrading to a newer protocol version may make older readers unable to open the table.

Key Connections

enables Lakehouse Architecture — the spec that makes Delta Lake ACID possible
solves Schema Evolution — schema enforcement in the transaction log
Delta Lake depends_on Delta Lake Protocol
scoped_to Table Formats, Lakehouse

Definition

What it is

A specification for ACID transaction logs over Parquet files on object storage. Defines how writes, deletes, and schema changes are recorded in a JSON-based transaction log stored alongside data files.

Why it exists

To bring database-like reliability to data lakes. The Delta protocol ensures that concurrent readers and writers see consistent table state, even on eventually consistent storage, by serializing changes through a commit log.

Primary use cases

ACID-compliant data lake tables, streaming + batch unification on S3, audit-trail via transaction log history.

Recent developments

Latest signals

Protocol versioning in practice: reader versions 1–3, writer versions 2–7, and table features as the fine-grained successor. A Delta table declares separate minReaderVersion and minWriterVersion values in its transaction log; reader version 3 plus writer version 7 unlock table features, which let a client adopt individual capabilities (deletion vectors, column mapping) without implementing everything from earlier protocol versions. Most protocol upgrades are irreversible — enabling a feature can permanently cut off older readers and writers — so treat upgrades like schema migrations, not config flips. Per Delta Lake feature compatibility and protocols (Databricks docs) and How does Delta Lake manage feature compatibility? (delta.io).
How the log coordinates concurrent writers: optimistic concurrency control. A writer records the table version it started from, does its work, then attempts to commit; if another writer landed first, the failed writer re-validates its operation against the new table state and retries — the mechanism that delivers serialized ACID commits on object storage without locks. Per Diving Into Delta Lake: Unpacking the Transaction Log (Databricks) and Delta Lake Transaction Log: How It Works (Conduktor).
The move from cumulative protocol versions to Table Features. The protocol transitioned away from cumulative version requirements: an implementation can now cherry-pick which table features it supports and jump straight to Writer Version 7 without carrying every feature from the versions in between. Per Delta Lake's transaction log protocol and its implementations (delta.io).
The canonical spec lives at PROTOCOL.md. The authoritative specification for the Delta Transaction Protocol — file layout, action types, and the protocol versions each feature requires — is maintained in the main repository and is what non-Spark implementations build against. Per delta/PROTOCOL.md (delta-io/delta).

Connections 6

Outbound 4

scoped_to2

Table Formats Lakehouse

enables1

Lakehouse Architecture

solves1

Schema Evolution

Inbound 2

depends_on1

Delta Lake

competes_with1

Apache Hudi Spec

Resources 4

SpecHigh

github.com/delta-io/delta/blob/master/PROTOCOL.md

The formal Delta Lake transaction log protocol specification defining the JSON action format, commit rules, checkpointing, schema enforcement, and time travel semantics.

DocsHigh

docs.delta.io/latest/index.html

Official Delta Lake documentation covering usage with Spark, API reference, table utilities, and operational guidance.

GitHubHigh

github.com/delta-io/delta

Canonical open-source repository for the Delta Lake project, containing the reference Spark-based implementation.

GitHubMedium

github.com/delta-incubator/delta-kernel-rs

Delta Kernel in Rust provides a standalone, engine-agnostic implementation of the Delta protocol, important for the multi-engine ecosystem.