Object Listing Performance
The slowness and cost of listing large numbers of objects in S3's flat namespace using prefix-based scans. Paginated at 1,000 objects per request.
Summary
The slowness and cost of listing large numbers of objects in S3's flat namespace using prefix-based scans. Paginated at 1,000 objects per request.
Object listing is the hidden bottleneck in S3 operations. Partition discovery, garbage collection, and table snapshots all start with listing — and at millions of objects, LIST calls dominate job startup time. Originates from: **S3 API**.
- S3 prefixes are not directories. A prefix scan does not benefit from directory-like structure — it is a linear scan filtered server-side.
- S3 Inventory (an offline listing report) is often better than real-time LIST for large-scale enumeration. But Inventory has a 24-48 hour delay.
- AWS S3
constrained_byObject Listing Performance — inherent API limitation - DuckDB, Trino
constrained_byObject Listing Performance — query engines pay the listing cost - Table formats reduce listing dependency by maintaining manifests, but metadata itself must be listed
scoped_toS3, Object Storage
Definition
The slowness and cost of listing large numbers of objects in S3's flat namespace using prefix-based scans.
Recent developments
- ListObjectsV2 returns up to 1,000 objects per request — hard limit. Listing 1M objects = 1,000 sequential round-trips minimum. Each call returns a ContinuationToken for pagination of the next page. The per-call cap is a hard S3 API limit. Per AWS docs — ListObjectsV2 API.
- Best practice: set MaxKeys lower than 1000 for memory/processing efficiency. While the maximum MaxKeys is 1000, setting a lower value reduces memory + processing time per call. Especially true when total object count is large and you want to start processing results during pagination rather than waiting for full pages. Per W3 Tutorials — AWS S3 List Objects pagination.
- S3 partition-by-prefix is the canonical mitigation. S3 automatically partitions buckets based on key prefixes to spread load across servers. Distributing keys across multiple prefixes multiplies the effective concurrency limit — the dominant 2026 pattern for high-LIST-throughput workloads is prefix-shard your keyspace, then parallel-LIST. Per OneUptime — S3 Prefixes and Partitioning for Performance.
- Standard S3 LIST is prefix-optimized but flat-namespace. ListObjectsV2 is already optimized for prefix-based listing (returns objects matching a prefix efficiently), but the flat namespace + 1000-per-call cap remains the structural ceiling. Per AWS docs — ListObjectsV2.
- Directory Buckets (S3 Express One Zone) use a hierarchical namespace instead — addresses this differently. Directory buckets in S3 Express One Zone restructure the namespace hierarchically, fundamentally changing the LIST cost profile for the hot tier — the per-call 1000-cap remains but the per-directory volume is bounded by directory size rather than full-bucket pagination. Per AWS docs — Differences for directory buckets.
- Third-party S3 implementations sometimes ship LIST bugs — known issue class. S3-compatible providers (SeaweedFS, others) have shipped ListObjectsV2 pagination bugs that manifest at multi-thousand object scale. Operators evaluating S3-compatible backends should stress-test LIST with >10K objects before adopting. Per GitHub — SeaweedFS issue #3166 ListObjectsV2 pagination broken.
Connections 9
Outbound 2
scoped_to2Inbound 7
Resources 3
Official AWS API reference for ListObjectsV2, documenting the 1,000-object-per-request limit and pagination mechanisms that constrain listing performance.
AWS's official performance design patterns covering S3 Inventory as an alternative to listing, prefix parallelization, and caching strategies for large-scale object enumeration.
Deep engineering investigation into why S3 ListObjects can take 120+ seconds, revealing how delete markers and versioning cause severe performance degradation.