Separation of Storage and Compute
The design pattern of keeping data in S3 while running independent, elastically scaled compute engines against it.
Summary
The design pattern of keeping data in S3 while running independent, elastically scaled compute engines against it.
This is the foundational architectural principle of the S3 ecosystem. Every query engine, table format, and data pipeline in this index assumes storage and compute are separate — data stays in S3, compute spins up and down on demand.
- Separation of storage and compute does not mean "no local storage." Caching, spill-to-disk, and local indexes are still used — the principle is that the source of truth is in S3.
- Network latency between compute and S3 is the fundamental trade-off. Every query pays the cost of reading over HTTP instead of local disk.
depends_onS3 API — the interface that enables decouplingsolvesVendor Lock-In — swap compute engines without moving dataconstrained_byCold Scan Latency, Egress Cost — the costs of network-based data access- ClickHouse
implementsSeparation of Storage and Compute scoped_toS3, Object Storage
Definition
The design pattern of keeping data exclusively in object storage (S3) while running independent, elastically scaled compute engines against it. Compute and storage scale independently.
Coupled storage-and-compute systems (traditional databases, HDFS with co-located compute) force you to scale both together. Decoupling allows you to pay for storage at S3 prices and spin compute up or down on demand.
Elastic analytics (scale query engines independently of data volume), multi-engine access (multiple query engines read the same S3 data), cost optimization.
Recent developments
- Databricks Lakebase GA on AWS (February 3, 2026) — serverless Postgres with compute-storage separation. Lakebase extends the lakehouse pattern into OLTP: serverless PostgreSQL built on Neon's compute-storage-separation architecture, integrated with Unity Catalog governance + Databricks analytics. The pattern reaches into transactional workloads. Per Databricks Blog — A New Era of Databases: Lakebase and Medium — From Lakehouse to Lakebase: Why Databricks Buying Neon Changes Everything.
- "Once WAL can be shipped over a network rather than written to disk, compute becomes stateless and replaceable." The architectural insight that makes Lakebase work — decoupling the database write-ahead log from local disk. Generalizes the lakehouse stateless-compute pattern to transactional databases. Per The Build — Postgres Goes to the Lake, Two Ways (April 2026).
- Iceberg + Lakebase shared-storage architecture. Predictions written to Iceberg tables appear in Postgres through the Lakebase layer — applications consume them in real time. OLTP + OLAP share the same underlying storage layer, eliminating data movement + duplication. Per Databricks Blog — Lakebase.
- Onehouse LakeBase: independent vendor offering for the same architectural pattern. Onehouse announced its own LakeBase product — "database speeds on the lakehouse" — confirming the architectural pattern as a market category, not a single-vendor pitch. Per Onehouse — Announcing Onehouse LakeBase: Database Speeds Finally on the Lakehouse.
- 2026 Moonfire framing: "The Lakehouse Era" — open storage + stateless compute is the default. Moonfire's 2026 retrospective frames the era: lakehouse architecture has fully arrived; the question is no longer "should we adopt this" but "what's our stack underneath?" Per Moonfire — The Lakehouse Era: How Open Storage and Stateless Compute Are Reshaping Data Infrastructure.
- Decoupling Postgres compute-storage at the architectural level published as analysis. Independent analysis of Lakebase + Neon-style architectures formalizes the compute-storage-decoupling pattern for the Postgres community — the lakehouse pattern's reach into traditional database designs. Per Medium — Decoupling Compute and Storage in Postgres: Architectural Implications of Databricks Lakebase (Feb 2026).
Connections 10
Outbound 6
scoped_to2depends_on1solves1constrained_by2Inbound 4
Resources 3
Snowflake's official architecture documentation describing the three-layer design (storage, compute, cloud services) that pioneered commercial separation of storage and compute.
Databricks' architecture documentation showing how Spark clusters run separately from data on S3/ADLS/GCS object storage.
Databricks glossary entry explaining how the lakehouse pattern depends on decoupled storage and compute.