Small File I/O Storm

The dominant performance pathology in S3-based data systems — a workload pattern where millions of small objects (typically <1 MB each) produce a per-request-latency-dominated I/O profile that defeats S3's throughput-oriented design. Each S3 LIST returns at most 1,000 objects (so listing 1M objects = 1,000 round-trips); each GET pays the same per-request HTTP+TLS+S3-routing overhead regardless of object size; and analytical query engines must open each data file individually before they can read it. Net effect: a 582K-small-file Athena query measured **40 seconds**; the same data compacted to 336 files of 247 MB each ran in **9.7 seconds** — a 75% reduction at the engine level alone.

S3 is purpose-built to optimize aggregate throughput on large objects (multi-megabyte reads), not single-object latency. Every object operation incurs a fixed cost (HTTP request, signature verification, S3 internal lookup, network round-trip) that's effectively constant whether the payload is 1 KB or 100 MB — so small objects pay the same overhead while delivering tiny payloads, destroying the throughput-per-request ratio. The pattern emerges naturally from streaming-ingestion pipelines (Kafka → S3 sink, IoT telemetry, CDC events) that produce many small objects per minute without any compaction step.

*(as a pain point — this is what triggers it:)* streaming ingestion into S3 lakehouse tables without compaction, log/event/telemetry pipelines that micro-batch into S3 every few seconds, CDC pipelines emitting per-row delete files, AI training pipelines writing per-sample checkpoint metadata, and any workload where the natural object emission rate creates millions of small files before compaction can run.