Lack of Atomic Rename
Summary
What it is
The S3 API has no atomic rename operation. Renaming requires copy-then-delete — a two-step, non-atomic process.
Where it fits
This limitation is the root cause of table format commit complexity on S3. Table formats need atomic commits to maintain consistency, and the workarounds (lock files, DynamoDB, conditional writes) add complexity and failure modes. Originates from: **S3 API**.
Misconceptions / Traps
- AWS S3 added conditional writes (If-None-Match) which help but do not fully replace atomic rename. Not all S3-compatible stores support this.
- Each table format handles this differently: Delta uses DynamoDB log stores, Iceberg uses metadata pointer files, Hudi uses markers. Know your format's approach.
Key Connections
- AWS S3, MinIO, Apache Iceberg, Delta Lake
constrained_byLack of Atomic Rename - Lakehouse Architecture
constrained_byLack of Atomic Rename — commits are complex on S3 - Originates from S3 API — fundamental protocol limitation
scoped_toS3
Definition
What it is
The absence of an atomic rename operation in the S3 API. Renaming an object requires copying it to a new key and then deleting the original — a two-step, non-atomic process.
Relationships
Outbound Relationships
scoped_toInbound Relationships
Resources
Delta Lake's official blog explaining how S3DynamoDBLogStore overcomes S3's lack of atomic rename by using DynamoDB for conditional put-if-absent semantics.
Databricks documentation explicitly listing Delta Lake limitations on S3 caused by the absence of atomic rename and put-if-absent operations.
Delta-rs documentation showing how the Rust-based Delta Lake implementation handles S3 writes with an external locking provider to compensate for missing atomic rename.