The Local-First S3 Index for LLM Data Infrastructure
— 296 concepts · 1381 relationships · 40 guidesEach technology, standard, and architecture in the index belongs to one or more topics — the conceptual anchors that define the S3 / AI-memory-infrastructure ecosystem. The seven topics added in the May 16, 2026 wave are highlighted.
Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this e...
The storage paradigm of flat-namespace, HTTP-accessible binary objects with metadata. Data is addressed by bucket and...
The category of specifications (Iceberg, Delta, Hudi) that bring table semantics — schema, partitioning, ACID transac...
The intersection of large language models and S3-centric data infrastructure. Scoped strictly to cases where LLMs ope...
The convergence of data lake storage (raw files on object storage) with data warehouse capabilities — ACID transactio...
The practice of building and querying vector indexes over embeddings derived from data stored in S3.
The emerging tier of persistent, object-storage-backed memory architecture sitting between GPU HBM and cold S3 — the ...
Using S3 as the central data layer for machine learning workflows: storing training data, model checkpoints, feature ...
The discipline of maintaining catalogs, schemas, statistics, and descriptive information about objects and datasets s...
The pattern of storing raw, heterogeneous data in object storage for later processing. Data arrives in its original f...
The practice of deploying S3-compatible object storage on infrastructure that is fully controlled by a specific organ...
The layer of standardized orchestration fabrics, communication protocols, model gateways, and agent runtimes that sit...
Deploying S3-compatible object storage at geographically distributed edge locations with synchronization to a central...
The architectural shift toward minimizing data movement between storage and inference compute — placing computation a...
The compliance, audit, lineage, and retention discipline applied to persistent AI memory — extending traditional data...
The set of technologies eliminating CPU bounce-buffers between object storage and GPU memory — establishing direct me...
Techniques for tracking and managing changes to datasets stored in object storage over time, including snapshots, bra...
A purpose-built storage tier designed for single-digit millisecond latency, using a directory-based namespace within ...
Kubernetes-native provisioning and management of S3 buckets using operators, the Container Object Storage Interface (...
The orchestration of memory and shared state across multi-agent environments — the architectural pattern that enables...
A design philosophy that treats object metadata as a first-class, queryable resource rather than an afterthought. Ena...
The ability to query a dataset as it existed at a previous point in time by leveraging immutable snapshots and metada...
The discipline of building production retrieval systems that go beyond basic Retrieval-Augmented Generation (RAG) — o...
I run local AI. Why do I care about S3?
Guided path from local inference to the S3 storage ecosystem — storage, formats, retrieval, and the tradeoffs that matter.
Architectural shifts as they happen. Each post anchors on a pre-existing pain point and walks through what changed.
When AI Memory Became an Architecture: KV-Cache Persistence, MCP, and the Night S3 Got Its Memory Tier
Stateful agents needed stateful storage. Tonight 33 new nodes joined the index: Mem0, Zep, LMCache, SGLang, Mooncake, MCP, cuObject, BlueField-4, Animesis CMA. The architectural thesis: AI memory infrastructure crystallized around object storage in 2026, and the persistence layer the industry settled on is S3.
The POSIX Gap is Closing: How S3 Quietly Became a File System
The May 7 post was about why storage suddenly matters again for AI workloads. This one is about how the access model itself evolved. After a decade of failed FUSE clients trying to bolt POSIX semantics onto S3, the storage service finally absorbed the operations the filesystem world expected — and a string of 2020-2026 changes (strong consistency, Express One Zone, directory buckets, S3 Tables, GPUDirect Storage) made S3 Files possible. Plus the parallel story in China: Aliyun CPFS+OSS, Huawei OBS+MindSpore, the same shape drawn three different ways.
When the AI Stack Became an I/O Stack: S3 Vectors GA, Real-Time Lakehouses, and the May 2026 Storage Rewrite
Amazon S3 Vectors hit GA at 20 trillion vectors per bucket. Apache Paimon is doing 40 million rows per second at ByteDance. Aliyun OSS embedded similarity search directly in the storage control layer. Three independent signals, one architectural truth: 2026's AI bottleneck is no longer compute — it's I/O, and the storage layer is being rewritten to absorb it.
How S3 Shapes Lakehouse Design
Every lakehouse architecture sits on object storage — almost always S3 or an S3-compatible store. But S3 is not a database, and its constrai...
7Choosing a Table Format — Iceberg vs. Delta vs. Hudi
The three major open table formats — Apache Iceberg, Delta Lake, and Apache Hudi — all solve the same fundamental problem: adding transactio...
2Small Files Problem — Why It Exists and the Common Mitigations
A dataset with 10 million 10KB files performs worse on S3 than the same data in 100 files of 1GB each. The small files problem is the most c...
4Where DuckDB Fits (and Where It Doesn't)
Engineers encounter S3-stored data constantly — Parquet files in data lakes, Iceberg tables in lakehouses, ad-hoc exports. Historically, exp...