Pain Point

Request Amplification

The phenomenon where a single logical operation (e.g., one SQL query, one table commit) generates a disproportionately large number of S3 API requests due to metadata reads, file listing, manifest parsing, and data file access patterns.

6 connections 3 resources

Summary

What it is

Where it fits

Request amplification is the root cause of unexpected S3 costs and throttling in lakehouse workloads. An Iceberg query that reads one row may issue hundreds of S3 GETs: metadata file, manifest list, manifest files, data files. Understanding and controlling this amplification is critical for cost and performance.

Misconceptions / Traps

Request amplification is not linear with data size. It scales with the number of metadata files, manifests, and data files — which is a function of table history and write frequency, not data volume.
S3 request throttling (503 SlowDown) due to request amplification appears as query latency, not as an explicit error in many query engines. Retries mask the throttling.
Caching metadata locally (manifest caching in Iceberg, metadata caching in Trino) dramatically reduces request amplification but introduces cache invalidation complexity.

Key Connections

scoped_to S3, Table Formats — S3 request multiplication in lakehouse operations
amplifies Request Pricing Models — more requests mean higher costs
amplifies Cold Scan Latency — metadata fetch latency adds up
solves Manifest Pruning — pruning reduces manifest read requests

Definition

What it is

The phenomenon where a single logical operation (one query, one compaction job) generates a disproportionately large number of S3 API calls, amplifying both latency and cost beyond what the logical operation warrants.