Lack of Atomic Rename
The S3 API has no atomic rename operation. Renaming requires copy-then-delete — a two-step, non-atomic process.
Summary
The S3 API has no atomic rename operation. Renaming requires copy-then-delete — a two-step, non-atomic process.
This limitation is the root cause of table format commit complexity on S3. Table formats need atomic commits to maintain consistency, and the workarounds (lock files, DynamoDB, conditional writes) add complexity and failure modes. Originates from: **S3 API**.
- AWS S3 added conditional writes (If-None-Match) which help but do not fully replace atomic rename. Not all S3-compatible stores support this.
- Each table format handles this differently: Delta uses DynamoDB log stores, Iceberg uses metadata pointer files, Hudi uses markers. Know your format's approach.
- AWS S3, MinIO, Apache Iceberg, Delta Lake
constrained_byLack of Atomic Rename - Lakehouse Architecture
constrained_byLack of Atomic Rename — commits are complex on S3 - Originates from S3 API — fundamental protocol limitation
scoped_toS3
Definition
The absence of an atomic rename operation in the S3 API. Renaming an object requires copying it to a new key and then deleting the original — a two-step, non-atomic process.
Recent developments
- S3 Express One Zone supports RenameObject API (June 2025). First AWS S3 storage class with native atomic rename — RenameObject API lets you rename existing objects atomically (single operation) without data movement. Operation typically completes in milliseconds regardless of object size. Per AWS What's New — S3 Express One Zone atomic renaming.
- RenameObject pricing matches PUT/COPY/POST/LIST. Same per-1,000-request pricing as the other S3 Express One Zone operations — no data upload or retrieval charges with the RenameObject API. Significant cost savings vs the traditional copy-and-delete pattern (which incurred both PUT + DELETE costs + intermediate-state visibility risk). Per AWS What's New — RenameObject pricing.
- RenameObject is S3 Express One Zone only — standard S3 still requires copy-and-delete. Critical limitation: the new atomic-rename API is only available for S3 Express One Zone (directory buckets). Standard S3 buckets still require the copy-and-delete workaround for renames, with all the original atomicity hazards. Per AWS docs — Renaming objects in directory buckets.
- Copy-and-delete remains the standard-S3 default — 5GB single-atomic limit. For standard S3, you can create a copy of an object up to 5 GB in a single atomic operation; objects above 5GB require multi-part copy. Cross-region copies incur bandwidth charges on top. Per AWS docs — Copying, moving, and renaming objects.
- Table-format engines have built around the limitation — but at architectural cost. Iceberg + Delta + Hudi use catalog-mediated atomic commits to work around the lack of atomic rename. The S3 Express One Zone RenameObject API removes the workaround need for hot-tier tables but doesn't help the much larger volume of standard-S3 tables. Per Iceberg architecture references on llms3.com.
- Google Cloud Storage offers a similar limitation + workaround. GCS docs cover the parallel copy/rename/move limitation in their object model — the S3-style limitation is a property of object storage as a category, not AWS-specific. Per Google Cloud — Copy, rename, and move objects.
Connections 9
Outbound 1
scoped_to1Inbound 8
constrained_by5solves2bypasses1Resources 3
Delta Lake's official blog explaining how S3DynamoDBLogStore overcomes S3's lack of atomic rename by using DynamoDB for conditional put-if-absent semantics.
Databricks documentation explicitly listing Delta Lake limitations on S3 caused by the absence of atomic rename and put-if-absent operations.
Delta-rs documentation showing how the Rust-based Delta Lake implementation handles S3 writes with an external locking provider to compensate for missing atomic rename.