Guides

36 cross-cutting guides anchored to the node graph

Object Storage Foundations 7
Essential 1
Guide 1

How S3 Shapes Lakehouse Design

Every lakehouse architecture sits on object storage — almost always S3 or an S3-compatible store. But S3 is not a database, and its constrai...

5 nodes 7 sources
Essential 8
Guide 8

Egress, Lock-In, and the Case for S3-Compatible Alternatives

AWS S3 egress pricing and proprietary feature creep create a gravitational well: data flows in cheaply but flows out expensively. For organi...

5 nodes 9 sources
10
Guide 10

Choosing an S3-Compatible Provider Beyond AWS

The S3 API has become the de facto standard for object storage, but AWS S3 is no longer the only serious option. Cloudflare R2 offers zero e...

3 nodes 9 sources
19
Guide 19

The Post-MinIO Landscape — Self-Hosted S3 Branches Out

For nearly a decade, "self-hosted S3" meant MinIO. It was the default answer — simple, fast, single-binary, open-source. The February 2026 a...

5 nodes 8 sources
29
Guide 29

Zero-Egress Architecture — Multi-Cloud Without the Bandwidth Tax

Egress fees dominate storage TCO for high-bandwidth workloads: multi-cloud AI training, edge inference, CDN origins, and cross-region analyt...

4 nodes 3 sources
30
Guide 30

Small Object Storage at Scale — Overcoming the Latency Tax

Standard S3 imposes per-request latency overhead regardless of object size: a 1 KB GET takes the same round-trip time as a 1 MB GET, but ret...

4 nodes 3 sources
32
Guide 32

The Local-First S3 Data Ecosystem — Architecting Resilient AI Pipelines for Constrained Environments

Engineers building AI pipelines on single-node servers, small Docker clusters, or prosumer-grade hardware need to replicate the functionalit...

5 nodes 18 sources
Table Formats & Lakehouse 8
3
Guide 3

Why Iceberg Exists (and What It Replaces)

Before Iceberg, querying data on S3 meant pointing a Hive Metastore at a directory of Parquet files and hoping for the best. There were no t...

5 nodes 7 sources
Essential 7
Guide 7

Choosing a Table Format — Iceberg vs. Delta vs. Hudi

The three major open table formats — Apache Iceberg, Delta Lake, and Apache Hudi — all solve the same fundamental problem: adding transactio...

5 nodes 8 sources
17
Guide 17

Real-Time Lakehouse: Paimon vs. Hudi

Building a real-time lakehouse on S3 requires a table format optimized for high-frequency writes and low-latency reads. Apache Hudi and Apac...

5 nodes 6 sources
Essential 1
Guide 1

How S3 Shapes Lakehouse Design

Every lakehouse architecture sits on object storage — almost always S3 or an S3-compatible store. But S3 is not a database, and its constrai...

5 nodes 7 sources
20
Guide 20

The True Cost of Managed Iceberg — S3 Tables vs. Self-Managed Compaction

AWS S3 Tables automates Iceberg table maintenance — compaction (binpack, sort, z-order), snapshot expiration, and orphan file cleanup — but ...

4 nodes 4 sources
21
Guide 21

Iceberg v3 in Production — Deletion Vectors and Row Lineage

Iceberg v2 positional delete files cause significant read amplification during query execution: engines must join data files against delete ...

5 nodes 5 sources
22
Guide 22

Table Format Interoperability — XTable, UniForm, and the End of Format Lock-In

Organizations that use multiple table formats — Iceberg for analytics, Delta for Spark-native pipelines, Hudi for CDC — face a metadata frag...

4 nodes 3 sources
34
Guide 34

DuckLake and the Future of Lakehouse Metadata

Every open table format — Iceberg, Delta Lake, Hudi — stores its metadata as files on S3. Iceberg writes Avro manifests and JSON table metad...

5 nodes 6 sources
Metadata / Catalog / Control Plane 4
15
Guide 15

The Great Catalog Migration

The "Metastore Era" — dominated by the Hive Metastore (HMS) — is ending. As Iceberg overtakes Hive-style tables, organizations must migrate ...

5 nodes 6 sources
13
Guide 13

AWS S3's New Native Features — Tables, Vectors, Metadata, Express

AWS is expanding S3 from pure object storage into a data platform. S3 Tables provides managed Apache Iceberg tables with automatic compactio...

4 nodes 9 sources
25
Guide 25

The Catalog Wars — Apache Polaris vs. Unity Catalog

The metadata catalog has replaced the table format as the critical vendor lock-in layer. While Iceberg, Delta, and Hudi compete at the file ...

4 nodes 4 sources
31
Guide 31

Credential Vending in Modern Data Lakes

Distributing static IAM access keys to distributed compute clusters — Spark executors, Trino workers, Flink task managers — is a security li...

4 nodes 4 sources
Query Engines & Analytics 2
Essential 4
Guide 4

Where DuckDB Fits (and Where It Doesn't)

Engineers encounter S3-stored data constantly — Parquet files in data lakes, Iceberg tables in lakehouses, ad-hoc exports. Historically, exp...

4 nodes 4 sources
27
Guide 27

SIMD and the C++ Query Engine Revolution

Java-based query engines (Trino, Spark) dominate the lakehouse ecosystem but impose structural performance ceilings: JVM garbage collection ...

4 nodes 2 sources
Performance / Layout / Compaction 4
Essential 2
Guide 2

Small Files Problem — Why It Exists and the Common Mitigations

A dataset with 10 million 10KB files performs worse on S3 than the same data in 100 files of 1GB each. The small files problem is the most c...

5 nodes 6 sources
18
Guide 18

The Single-Digit Millisecond S3

The 2025-2026 S3 ecosystem has shattered the assumption that object storage is inherently slow. S3 Express One Zone delivers single-digit mi...

5 nodes 6 sources
30
Guide 30

Small Object Storage at Scale — Overcoming the Latency Tax

Standard S3 imposes per-request latency overhead regardless of object size: a 1 KB GET takes the same round-trip time as a 1 MB GET, but ret...

4 nodes 3 sources
32
Guide 32

The Local-First S3 Data Ecosystem — Architecting Resilient AI Pipelines for Constrained Environments

Engineers building AI pipelines on single-node servers, small Docker clusters, or prosumer-grade hardware need to replicate the functionalit...

5 nodes 18 sources
Governance / Security / Compliance 3
12
Guide 12

Ransomware Protection and Immutable Backups on Object Storage

Ransomware attacks increasingly target backup infrastructure specifically — encrypting or deleting backups before encrypting production data...

5 nodes 6 sources
28
Guide 28

Defending S3 Against SSE-C Encryption Hijacking

Cloud-native ransomware has evolved beyond data deletion. Attackers with compromised IAM credentials use the S3 CopyObject API with SSE-C (S...

4 nodes 4 sources
31
Guide 31

Credential Vending in Modern Data Lakes

Distributing static IAM access keys to distributed compute clusters — Spark executors, Trino workers, Flink task managers — is a security li...

4 nodes 4 sources
AI & Retrieval Over Structured Data 8
Essential 5
Guide 5

Vector Indexing on Object Storage — What's Real vs. Hype

Vector databases and semantic search are heavily marketed features in the AI ecosystem. For engineers building on S3, the question is practi...

7 nodes 8 sources
6
Guide 6

LLMs over S3 Data — Embeddings, Metadata, and Local Inference Constraints

LLMs can extract value from S3-stored data — generating embeddings, extracting metadata, classifying documents, inferring schemas, and trans...

6 nodes 10 sources
16
Guide 16

S3 Vectors vs. Dedicated DBs

Amazon S3 Vectors (GA 2025) integrates vector storage and similarity search directly into S3, challenging the assumption that RAG pipelines ...

5 nodes 6 sources
9
Guide 9

Object Storage for AI/ML Training Pipelines

AI/ML training workloads are becoming the dominant consumers of object storage bandwidth. A single LLM training run may read tens of terabyt...

5 nodes 8 sources
23
Guide 23

Billion-Scale Vector Search on S3 — Decoupling Compute and Storage

Storing full-precision HNSW graphs in RAM becomes economically unviable beyond approximately 100 million vectors — a billion 768-dimensional...

4 nodes 4 sources
24
Guide 24

The Lance Format — ML-Native Storage Beyond Parquet

Apache Parquet organizes data into monolithic row groups optimized for sequential columnar scans. This layout causes severe I/O bottlenecks ...

5 nodes 4 sources
32
Guide 32

The Local-First S3 Data Ecosystem — Architecting Resilient AI Pipelines for Constrained Environments

Engineers building AI pipelines on single-node servers, small Docker clusters, or prosumer-grade hardware need to replicate the functionalit...

5 nodes 18 sources
33
Guide 33

Choosing a Vector Database for S3 Workloads

The vector database market has fragmented into three distinct architectural tiers, each with a fundamentally different relationship to S3-co...

7 nodes 8 sources
Self-Hosted / Hybrid / Edge S3 5
14
Guide 14

Surviving the MinIO Archive

In early 2026, the MinIO community repository was archived, effectively ending the open-source era of the most widely deployed S3-compatible...

5 nodes 6 sources
11
Guide 11

Multi-Site Replication and Geo-Distributed Object Storage

Operating object storage across multiple geographic sites introduces problems that single-site deployments never encounter. Engineers deploy...

5 nodes 8 sources
19
Guide 19

The Post-MinIO Landscape — Self-Hosted S3 Branches Out

For nearly a decade, "self-hosted S3" meant MinIO. It was the default answer — simple, fast, single-binary, open-source. The February 2026 a...

5 nodes 8 sources
32
Guide 32

The Local-First S3 Data Ecosystem — Architecting Resilient AI Pipelines for Constrained Environments

Engineers building AI pipelines on single-node servers, small Docker clusters, or prosumer-grade hardware need to replicate the functionalit...

5 nodes 18 sources
36
Guide 36

POSIX Compatibility on Object Storage — When You Need a Filesystem Over S3

S3 is a key-value store with HTTP semantics, not a filesystem. It lacks atomic rename, returns directory listings via paginated API calls, a...

5 nodes 5 sources
Ingestion & Streaming 3
17
Guide 17

Real-Time Lakehouse: Paimon vs. Hudi

Building a real-time lakehouse on S3 requires a table format optimized for high-frequency writes and low-latency reads. Apache Hudi and Apac...

5 nodes 6 sources
26
Guide 26

CDC Failure Modes — What Breaks When Streaming Database Logs to S3

Change Data Capture into a lakehouse is marketed as a turnkey path to "real-time analytics," but production CDC pipelines routinely fail fro...

4 nodes 4 sources
35
Guide 35

Python-Native Stream Processing — Bytewax vs. Flink for S3 Ingestion

Real-time ingestion into S3 lakehouses has traditionally meant Apache Flink — a distributed, stateful stream processor with mature Iceberg s...

4 nodes 7 sources
Cost / FinOps / Benchmarks 3
Essential 8
Guide 8

Egress, Lock-In, and the Case for S3-Compatible Alternatives

AWS S3 egress pricing and proprietary feature creep create a gravitational well: data flows in cheaply but flows out expensively. For organi...

5 nodes 9 sources
20
Guide 20

The True Cost of Managed Iceberg — S3 Tables vs. Self-Managed Compaction

AWS S3 Tables automates Iceberg table maintenance — compaction (binpack, sort, z-order), snapshot expiration, and orphan file cleanup — but ...

4 nodes 4 sources
29
Guide 29

Zero-Egress Architecture — Multi-Cloud Without the Bandwidth Tax

Egress fees dominate storage TCO for high-bandwidth workloads: multi-cloud AI training, edge inference, CDN origins, and cross-region analyt...

4 nodes 3 sources