The Local-First S3 Index for LLM Data Infrastructure
—438 concepts·1992 relationships·51 guidesEach technology, standard, and architecture in the index belongs to one or more topics — the conceptual anchors that define the S3 / AI-memory-infrastructure ecosystem, sorted by how connected they are.
Amazon's Simple Storage Service and the broader ecosystem of S3-compatible object storage. The root concept of this e...
The storage paradigm of flat-namespace, HTTP-accessible binary objects with metadata. Data is addressed by bucket and...
The emerging tier of persistent, object-storage-backed memory architecture sitting between GPU HBM and cold S3 — the ...
The category of specifications (Iceberg, Delta, Hudi) that bring table semantics — schema, partitioning, ACID transac...
The convergence of data lake storage (raw files on object storage) with data warehouse capabilities — ACID transactio...
The layer of standardized orchestration fabrics, communication protocols, model gateways, and agent runtimes that sit...
The intersection of large language models and S3-centric data infrastructure. Scoped strictly to cases where LLMs ope...
Using S3 as the central data layer for machine learning workflows: storing training data, model checkpoints, feature ...
The practice of building and querying vector indexes over embeddings derived from data stored in S3.
The discipline of maintaining catalogs, schemas, statistics, and descriptive information about objects and datasets s...
The compliance, audit, lineage, and retention discipline applied to persistent AI memory — extending traditional data...
The practice of deploying S3-compatible object storage on infrastructure that is fully controlled by a specific organ...
The pattern of storing raw, heterogeneous data in object storage for later processing. Data arrives in its original f...
The architectural shift toward minimizing data movement between storage and inference compute — placing computation a...
The set of technologies eliminating CPU bounce-buffers between object storage and GPU memory — establishing direct me...
Deploying S3-compatible object storage at geographically distributed edge locations with synchronization to a central...
Techniques for tracking and managing changes to datasets stored in object storage over time, including snapshots, bra...
The discipline of building production retrieval systems that go beyond basic Retrieval-Augmented Generation (RAG) — o...
A purpose-built storage tier designed for single-digit millisecond latency, using a directory-based namespace within ...
Kubernetes-native provisioning and management of S3 buckets using operators, the Container Object Storage Interface (...
The orchestration of memory and shared state across multi-agent environments — the architectural pattern that enables...
A design philosophy that treats object metadata as a first-class, queryable resource rather than an afterthought. Ena...
The ability to query a dataset as it existed at a previous point in time by leveraging immutable snapshots and metada...
I run local AI. Why do I care about S3?
Guided path from local inference to the S3 storage ecosystem — storage, formats, retrieval, and the tradeoffs that matter.
Architectural shifts as they happen. Each post anchors on a pre-existing pain point and walks through what changed.
When the Lakehouse Became an Acquisition Target
2026 is the year the buyers came for the open data stack. SAP closed Dremio at a disclosed ~€0.5B — the number is in an SEC filing, not a press release. Databricks bought its third security company to build the 'security lakehouse.' NetApp bought DataPelago to own the acceleration layer. The pattern across all three: incumbents are buying Iceberg-ecosystem assets, and the independent layer this index maps is consolidating in real time. Here is the scorecard, who is left, and why the economics chose this exact year.
When the Quota Became a Variable
We measured our own $20 inference plan dropping from ~2M tokens per session to ~700K — same plan, same price, no changelog. Then we pulled the industry-wide evidence: a documented 2025–2026 wave of quota re-denominations, peak-hour throttles admitted only after user telemetry caught them, and deprecations that silently rebill at 6× the rate. Also the counter-evidence, because the thesis only survives if it's tested: OpenAI uncapped text, Anthropic doubled limits in May, and DeepSeek's price floor keeps falling. The pattern isn't uniform gouging — it's bifurcation plus unit games. And it's the strongest argument yet for the stack this index maps.
When the Network Became the Storage Bottleneck
The pain point this index mapped in May — GPUs idle, waiting on object storage — just got its industry answer, and it isn't faster storage software. It's three competing network fabrics: Ultra Ethernet 1.0 with packet spraying, InfiniBand XDR at 800G per port, and RoCEv2 eating the middle. Every S3-to-GPU byte now crosses a wire with three possible futures, and the storage tier's ceiling is set by which one you pick.
Getting Data Into the Lakehouse — Choosing a CDC-to-Iceberg Path in 2026
The default way to move operational data into the lakehouse used to be a four-system pipeline: Debezium reads the database WAL, Kafka buffer...
48Choosing a Lakehouse Catalog — Polaris vs. Unity Catalog vs. Gravitino vs. Cloud-Native
In 2024 the catalog was an afterthought — somewhere to remember where the tables lived. By mid-2026 it is the **control plane** of the lakeh...
19The Post-MinIO Landscape — Self-Hosted S3 Branches Out
For nearly a decade, "self-hosted S3" meant MinIO. It was the default answer — simple, fast, single-binary, open-source. The February 2026 a...
37Picking an AI Memory Layer in 2026 — Mem0 vs. Zep vs. Build-Your-Own
The shift from stateless LLM inference to stateful, multi-agent systems forces a decision that didn't exist two years ago: where does agent ...