Architecture

Compaction

The background maintenance operation that merges many small data files into fewer, larger files within a table format (Iceberg, Delta, Hudi) to improve query performance and reduce S3 request overhead.

10 connections 3 resources

Summary

What it is

Where it fits

Compaction is the primary remedy for the small files problem in S3-based lakehouses. Streaming ingestion, CDC pipelines, and frequent batch writes all produce small files that degrade scan performance. Compaction rewrites those files into optimally sized Parquet files while preserving table semantics.

Misconceptions / Traps

Compaction is not free. It reads existing files from S3, merges them, writes new files, and updates metadata. This consumes compute, S3 GET/PUT requests, and temporary storage.
Running compaction too aggressively conflicts with active writers. In Iceberg, concurrent compaction and writes can cause commit conflicts requiring retry.
Compaction does not reduce data volume. It reorganizes files for efficiency but does not delete or deduplicate data. Storage usage may temporarily increase during compaction before old files are garbage-collected.

Key Connections

solves Small Files Problem — the primary purpose of compaction
solves Small Files Amplification — reduces metadata and request overhead
scoped_to Table Formats, S3 — operates within table format maintenance
used_by Apache Iceberg, Delta Lake, Apache Hudi — all formats provide compaction mechanisms

Definition

What it is

The process of rewriting many small data files on S3 into fewer, larger files to improve query performance, reduce metadata overhead, and lower API call costs — without changing the logical content of the table.

Why it exists

Streaming ingestion, CDC, and frequent small writes produce many small files on S3. Without periodic compaction, query engines must open thousands of files per query, inflating latency and S3 GET costs. Compaction restores optimal file sizes.

Primary use cases

Post-ingestion file consolidation in Iceberg/Delta/Hudi tables, scheduled maintenance for streaming lakehouse pipelines, metadata size reduction.

Recent developments

Latest signals

Three Iceberg compaction strategies: binpack / sort / z-order. Binpack is the default (combines files for basic optimization, fastest); sort organizes files by user-defined column order (clusters similar values for filter-query speedup); z-order interleaves binary representations of multiple columns into a single scalar (best for multi-dimensional / spatial queries). Per Dremio — Compaction in Apache Iceberg.
Amazon S3 added sort + z-order compaction (July 2025). AWS extended S3 Tables compaction beyond the default binpack to support sort + z-order — performance improvements of 3× or more observed depending on data layout + query patterns. Per InfoQ — Amazon S3 Sort + Z-Order Compaction.
Z-order delivers 2-level pruning: partition + multi-dimensional file skipping. Z-order skipping interleaves binary values of multiple columns so a single sort dimension drives multi-dimensional pruning. Combined with partition pruning, this is the optimal layout for production Iceberg tables with complex query patterns. Per Iceberg Lakehouse — Z-Order Clustering in Iceberg.
When to skip sort: frequent fast compaction needs binpack only. If you run frequent compaction that needs to complete quickly, binpack is the right choice — sort/z-order's clustering pass adds compute cost that doesn't pay back when files churn quickly. Per AWS Blog — Improve Iceberg Query Performance with Sort and Z-Order.
S3 Tables sort-order configuration: must be set on the table before compaction picks it up. Operators configuring S3 Tables for sort/z-order compaction must define the sort order at the table level; S3 Tables compaction will then use that order to cluster similar values during compaction. Per AWS re:Post — How to set sort order on S3 Tables.