Topic

S3

Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this entire index.

171 connections 4 resources

Summary

What it is

Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this entire index.

Where it fits

Every node in the index answers the question "How does this relate to S3?" S3 is not just a product — it is the API, the paradigm, and the ecosystem that the rest of the map is built around.

Misconceptions / Traps
  • S3 is not a filesystem. It has no directories, no atomic rename, and no POSIX semantics. Treating it like a filesystem causes subtle bugs.
  • "S3-compatible" does not mean identical. Consistency guarantees, performance characteristics, and feature coverage vary across providers.
Key Connections
  • Root topic — all other Topics connect inward via scoped_to
  • Object Storage scoped_to S3 — S3 is the dominant implementation of the object storage paradigm
  • S3 API scoped_to S3 — the HTTP interface that defines the ecosystem
  • AWS S3 scoped_to S3 — the origin and reference implementation

Definition

What it is

The conceptual root of this index. Encompasses Amazon S3, the ecosystem of S3-compatible object storage, the technologies built on top of it, and the engineering problems that arise at scale.

Why it exists

S3 created the object storage paradigm that underpins modern data infrastructure. Every node in this index answers the question: "How does this relate to S3?"

Connections 171

Inbound 171click to expand
scoped_to169
Topic14
Object StorageData LakeTable FormatsVector Indexing on Object StorageLLM-Assisted Data SystemsMetadata ManagementData VersioningDirectory Buckets / Hot Object StorageObject Storage for AI Data PipelinesKubernetes Object Provisioning & PolicyMetadata-First Object StorageGeo / Edge Object StorageTime TravelSovereign Storage
Technology58
AWS S3MinIOCephApache OzoneDuckDBTrinoClickHouseApache SparkLanceDBStarRocksApache FlinkS3 Express One ZoneAmazon S3 TablesAmazon S3 VectorsAmazon S3 MetadataSeaweedFSCloudflare R2Backblaze B2VAST DataDell ECSNetApp StorageGRIDPure Storage FlashBladeGarageOpenDALlakeFSRookGeeseFSApache PolarisApache GravitinoUnity CatalogApache XTableDelta UniFormApache PaimonFlink CDCEstuary FlowRustFSMarquezApache RangerS3 Bucket KeyWarpStreamApache DorisAWS Glue CatalogHive MetastoreDremioAthenaDebeziumDataFusionPolarsKafka Tiered StorageRedpandaProject NessieAirbyteSpark Structured StreamingVeloxdltOpenMetadataDataHubApache Atlas
Standard12
S3 APIApache ParquetApache ArrowORCApache AvroContainer Object Storage Interface (COSI)AWS Signature Version 4 (SigV4)Object Lock / WORM SemanticsOpenLineageS3 Directory BucketIceberg V3 SpecLance Format
Architecture49
Separation of Storage and ComputeHybrid S3 + Vector IndexOffline Embedding PipelineLocal Inference StackWrite-Audit-PublishTiered StorageGeo-Dispersed Erasure CodingNVMe-backed Object TierGPU-Direct Storage PipelineCache-Fronted Object StorageCheckpoint/Artifact Lake on Object StorageTraining Data Streaming from Object StorageFeature/Embedding Store on Object StorageOnline Embedding Refresh PipelineActive-Active Multi-Site Object ReplicationEdge-to-Core Object AggregationImmutable Backup Repository on Object StorageRansomware-Resilient Object Backup ArchitectureDeletion VectorLSM-tree on S3CompactionCDC into LakehouseRow / Column SecurityEncryption / KMSTenant IsolationRAG over Structured DataClustering / Sort OrderFile Sizing StrategyAudit TrailsPII TokenizationBatch vs StreamingEvent-Driven IngestionManifest PruningCompliance-Aware ArchitecturesBranching / TaggingAI-Safe ViewsStructured ChunkingBenchmarking MethodologyCapacity PlanningHybrid Metadata PatternsInteroperability PatternsNon-Blocking Concurrency ControlDecoupled Vector SearchPartitioningCredential VendingObject Lifecycle ManagementLakehouse for AI WorkflowsMultimodal Object StorageRedaction Layers
Pain Point27
Small Files ProblemCold Scan LatencyLegacy Ingestion BottlenecksHigh Cloud Inference CostObject Listing PerformancePartition Pruning ComplexityVendor Lock-InEgress CostS3 Consistency Model VarianceLack of Atomic RenameS3 Compatibility DriftDirectory Namespace / Listing BottlenecksGeo-Replication Conflict / DivergenceRetention Governance FrictionPolicy SprawlCold Retrieval LatencySmall Files AmplificationRequest Pricing ModelsCompression EconomicsData ResidencyRequest AmplificationCross-Region ConsistencyRead / Write AmplificationCache ROIPerformance-per-DollarZero-Egress EconomicsSSE-C Encryption Hijacking
Model Class3
Anomaly Detection ModelsCost Optimization ModelsPolicy Recommendation Models
LLM Capability6
Storage Class Lifecycle RecommendationCompatibility Test Case GenerationRansomware Pattern Detection from Object EventsCost Anomaly ExplanationPolicy Diff Review / Access AuditData Placement Recommendation
depends_on2
MilvusJuiceFS

Resources 4

DocsHigh
docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

The single authoritative reference for all S3 concepts, features, and operational guidance.

DocsHigh
docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html

The S3 REST API Reference defines every API operation, request/response format, and error code — the ground truth for any S3 integration.

DocsHigh
docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-per...

AWS's own performance design patterns covering partitioning, request rate scaling, and throughput optimization for production S3 workloads.

DocsHigh
aws.amazon.com/s3/

The S3 product landing page provides pricing tiers, storage class overviews, and feature announcements in a single consolidated view.