Small Files Problem
Too many small objects in S3 degrade query performance and increase API call costs. Each file requires a separate GET request, and S3 charges per-request.
Summary
Too many small objects in S3 degrade query performance and increase API call costs. Each file requires a separate GET request, and S3 charges per-request.
The small files problem is the most common performance issue in S3-based data systems. It affects every query engine, every table format, and every streaming pipeline that writes to S3.
- The threshold is not absolute. "Small" depends on the workload, but files under 100MB are generally problematic for analytics. Aim for 256MB-1GB per file.
- Compaction solves the problem retroactively but not proactively. Fix the root cause (writer parallelism, streaming micro-batches) in addition to running compaction.
- Apache Iceberg
solvesSmall Files Problem — via compaction - DuckDB, Trino, Apache Spark, Apache Flink
constrained_bySmall Files Problem — performance degradation - Medallion Architecture
constrained_bySmall Files Problem — each layer can produce small files scoped_toS3, Object Storage
Definition
The accumulation of too many small objects in S3, which degrades query performance and increases API call costs. Analytical queries must open each file individually, and S3 charges per-request.
Recent developments
- Root cause cohort: CDC + Kafka + continuous ETL all generate small files by design. Real-time pipelines are structural producers of small files — a Flink streaming job with 1-minute checkpoints creates 1-10 MB files per checkpoint, ~1,440 files per day. The "small files problem" is a symptom of "we adopted streaming"; the only mitigation is compaction on a discipline. Per Olake — How to Compact Apache Iceberg Tables: Small Files + Automation with Apache Amoro.
- Target file size: 128-512 MB Parquet (256 MB optimal). 2026 production reference: merge small files into target size 128-512 MB, with 256 MB as the optimal default. Below 128 MB → excessive metadata + planning latency; above 1 GB → reduced pruning effectiveness. Per DEV — The Apache Iceberg Small File Problem and Dremio — Compaction in Apache Iceberg.
- Dremio + Snowflake + AWS ship managed compaction in 2026. Cloud-platform-managed compaction is the new default — Dremio Open Catalog runs compaction + expiry + cleanup without user configuration; AWS S3 Tables ship managed compaction; Snowflake automates Iceberg-table compaction. The "you must engineer your own compaction DAG" era is closing for cloud-managed deployments. Per Iceberg Lakehouse — Maintaining Apache Iceberg Tables: Compaction, Expiry, Cleanup.
- Compaction cadence: daily for read-heavy / every few hours for streaming. 2026 cadence-tuning reference: read-heavy tables run compaction once daily off-peak; streaming/CDC tables run every few hours to keep accumulation bounded. The cadence is the load-bearing operational discipline that small-files-aware deployments tune first. Per Dremio — Compaction in Apache Iceberg.
- Apache Amoro automates compaction-as-a-service for open-source deployments. Open-source automation pattern: deploy Amoro alongside the catalog — watches tables + triggers compaction at configured thresholds. The OSS counterpart to AWS/Snowflake managed compaction. Per Olake — Apache Iceberg Small Files + Automation with Apache Amoro.
- Table-degradation framing: "how data lake table storage degrades over time." 2026 framing positions small-files as one of three concurrent degradation processes — small files, metadata bloat, catalog fragmentation. Production maintenance treats all three together; addressing only files leaves the other two compounding. Per Iceberg Lakehouse — How Data Lake Table Storage Degrades Over Time.
Connections 20
Outbound 2
scoped_to2Inbound 18
solves9constrained_by9Resources 2
Delta Lake's official blog explaining the OPTIMIZE command for compacting small files into ~1 GB targets, directly addressing the classic small files problem.
Databricks documentation on controlling data file size with auto compaction and optimized writes, the production solution for small files in Delta tables.