Tiered Storage
Moving data between hot, warm, and cold storage tiers based on access frequency. S3 itself offers tiering (Standard, Infrequent Access, Glacier).
Summary
Moving data between hot, warm, and cold storage tiers based on access frequency. S3 itself offers tiering (Standard, Infrequent Access, Glacier).
Tiered storage is the cost optimization layer for S3 data. It ensures frequently accessed data is fast and expensive while archival data is slow and cheap — a critical pattern for large data lakes where 80%+ of data is rarely accessed.
- Retrieval from cold tiers (Glacier, Deep Archive) has latency measured in minutes to hours. Do not tier data that might be needed for interactive queries.
- S3 Intelligent-Tiering automates tier transitions but has per-object monitoring charges. For predictable access patterns, explicit lifecycle rules are cheaper.
solvesEgress Cost — keeps hot data close to compute, cold data in cheap tiersconstrained_byVendor Lock-In — tiering policies and pricing are provider-specificscoped_toS3, Object Storage
Definition
The pattern of moving data between hot, warm, and cold storage tiers based on access frequency, with S3 or S3-compatible stores serving as one or more tiers.
Not all data is accessed equally. Frequently queried data benefits from fast (expensive) storage; archival data can reside in cheap, slow tiers. S3 itself offers tiering (Standard, Infrequent Access, Glacier), and cross-provider tiering adds further cost optimization.
Cost optimization for large data lakes, lifecycle management of S3-stored datasets, compliance archival, AI training infrastructure where the same dataset spans hot scratch (NVMe), warm checkpoints (HDD), cool training corpora (S3), and archive (Glacier/tape).
Recent developments
- S3 Intelligent-Tiering: 30-day IA + 90-day Archive Instant Access auto-transitions. S3 Intelligent-Tiering monitors access patterns and moves objects to Infrequent Access after 30 days of no access, Archive Instant Access after 90 days — without per-retrieval fees and without operational overhead. Three tiers: frequent / 40% cheaper IA / 68% cheaper rarely-accessed. Per AWS — S3 Intelligent-Tiering Storage Class and AWS Docs — How S3 Intelligent-Tiering Works.
- Glacier Instant Retrieval: $0.004/GB-month with millisecond access. Same throughput + ms-scale latency as S3 Standard, at 1/6 the cost — the tier for "rarely accessed but must be instantly available when needed" (regulatory archives, medical imaging, financial records). Per AWS — S3 Storage Classes.
- "Glacier's trap": $1/TB to store, $20K to retrieve large-scale archive. 2026 cost-awareness shift: Glacier Deep Archive at $1/TB-month is the cheapest cloud storage, but retrieval cost + retrieval latency make it a one-way-door for anything you might actually need. LeanOps' 2026 cost analysis names this as the most-misunderstood AWS pricing pattern. Per LeanOps — Glacier's Trap: $1/TB to Store, $20K to Retrieve.
- AI training shape: NVMe → HDD → S3 → tape becomes the new 4-tier reference. Netflix's published 5 PB NVMe → 100 PB HDD → 500 PB S3 → 2 EB tape architecture is the canonical reference shape. Request distribution inverts capacity distribution: NVMe holds 1-2% of capacity but absorbs ~60% of requests; tape holds 10-20% of capacity but <1% of reads. Reported ~$45M/year savings vs all-S3. Per project notes + Sedai — S3 Intelligent-Tiering Guide.
- S3 Intelligent-Tiering has no retrieval charges. Key differentiator vs Glacier classes — for unpredictable access patterns, Intelligent-Tiering is the safer default precisely because there's no surprise-cost from a retrieval spike. Per CloudFix — S3 Intelligent-Tiering Pricing 2026.
- Pattern consolidating: Intelligent-Tiering as ingestion default + Glacier classes for specific compliance-archive use cases. 2026 best-practice frame: don't pick a fixed storage class at upload time unless you have a specific compliance reason; default to Intelligent-Tiering and let the access-pattern monitor decide. Per CloudZero — 2026 Guide to Amazon S3 Pricing.
Connections 8
Outbound 5
Inbound 3
enables1related_to1Resources 3
Official AWS documentation of all S3 storage classes (Standard, Intelligent-Tiering, Glacier, Deep Archive), the canonical reference for S3-based tiered storage.
Official Apache Kafka documentation for tiered storage (KIP-405), which offloads older log segments to S3/HDFS while keeping recent data on local brokers.
Confluent's production-grade tiered storage documentation showing how Kafka integrates with S3 for virtually unlimited, low-cost retention.