Ten days ago we closed the July field report with the floor compounding — RDMA reached object storage, the query engines converged, the catalog became a control plane.1 This report is about what happened inside that control plane, and below it, in the pipes.
Three stories, one thread: the intermediate layers of the data stack are being asked to justify their existence, and most of them can't.
The catalog race tilted on a governance decision
In June we mapped two open catalogs racing for the Iceberg REST Catalog spec from opposite directions: Apache Polaris with enterprise security, Project Nessie with Git semantics — branches, tags, atomic multi-table commits.
Our read, as of July 2026: this race has tilted decisively toward Polaris — and here is the evidence, along with what would change our mind. Polaris graduated to an Apache Top-Level Project in February (100+ contributors, 2,800+ closed PRs), survived its first serious security reckoning — the four-CVE credential-vending cluster, patched in 1.4.12 — and converted the incident into a release train: 1.5.0 shipped BigQuery Metastore federation and the Ranger authorizer plugin Cloudera contributed with its adoption; 1.6.0 landed July 8 with structured STS session names on every vended credential.3 The community debate has moved on to semantic layers, lineage, and serving AI agents — next-surface arguments, not viability arguments.
Nessie, meanwhile, drew the assessment no project wants from two independent practitioner comparisons: not recommended for greenfield.4 Not because Git-for-data was wrong — because Nessie never vended credentials. Every engine connecting to it holds its own storage keys, which breaks the centralized-governance model that is the entire point of the catalog layer in 2026. The industry's revealed preference is blunt: credential vending and RBAC beat branching semantics, and the published expectation — which we share — is that Nessie's versioning ideas get absorbed into Polaris rather than winning on their own. We would revise this read if Nessie ships native credential vending, or if significant new greenfield deployments surface. Neither has, as of this writing.
The lesson generalizes past catalogs. In the control plane, security features are the product. Lakekeeper — the Rust catalog we added last wave — understood this from day one and just extended it: 0.12 ships exactly-once audit events, Cedar batch authorization, and a Generic Table API that puts Lance and Delta tables under the same credential-vending and soft-delete protections as Iceberg.5
The pipeline collapsed from four systems to one
The canonical CDC pipeline — Debezium reads the WAL, Kafka buffers it, Flink transforms it, Parquet lands on S3 — is being dismantled hop by hop.
The number that reframed the category: Supermetal, a single Rust process, snapshots a 375-million-row Postgres into Apache Iceberg in 13 minutes. Flink CDC takes 90–116 minutes on the same normalized single node — after aggressive tuning. Kafka Connect takes 120. Spark, 200+.6 The mechanism is worth understanding, because it's not magic: WAL events decode straight into Arrow record batches and write out as Parquet — columnar end-to-end, no serialize-to-JSON-then-deserialize tax — and the Iceberg writer changes behavior by pipeline phase, running append-only during snapshot and switching to Iceberg V3 deletion vectors for live updates.7
Flink isn't standing still — CDC 3.6.0 shipped PostgreSQL schema evolution and the VARIANT type, so upstream ALTER TABLE events now propagate into Iceberg automatically8 — and it keeps the distributed exactly-once high ground. But the burden of proof flipped. The question is no longer "why would you use a single process instead of a proper pipeline?" It's "why are you operating four distributed systems to move rows from Postgres to S3?"
The same flip is happening one layer up. Bytewax runs Python-native streaming at a 25× lower memory footprint than Flink — 4.1 GB against 97.9 GB for four equivalent pipelines, a 4.6× total-cost gap.9 RisingWave deletes the pipeline entirely for SQL-shaped workloads: declare a materialized view, and ingestion, incremental computation, and serving collapse into one Postgres-compatible system. Kafka's unbundling isn't a rebellion anymore. It's the default architecture question.
The lakehouse went real-time, and the two-engine pattern lost its reason
The strongest version of this consolidation story shipped in March and went mainstream this month: Real-Time Mode for Spark Structured Streaming, GA on Databricks with OSS support in the Spark 4.1 line.10
Microbatching imposed a latency floor — shrink the batch interval and fixed overheads (scheduling, planning, checkpointing to object storage) eat the budget. RTM's redesign attacks exactly that: epochs stretch to minutes with data flowing continuously through operators, reducers start consuming shuffle files the moment they exist, and blocking operators were rebuilt to emit continuously. Result: sub-100ms end-to-end, up to 92% faster than Flink on feature-computation workloads. Coinbase runs 250+ ML features on it at sub-100ms P99; DraftKings does live-betting fraud detection on transformWithState.10
The casualty isn't Flink's latency crown — it's the two-engine pattern and the logic drift it caused. Training features computed in Spark, served in Flink, subtly diverging: that was an accepted cost of the batch/streaming split. When one engine covers both at production latency, the drift stops being a cost anyone has to accept. Whether teams actually consolidate is a 2027 question — RTM is four months old, and Coinbase/DraftKings are launch references, not a movement. What's settled is that the justification for running two engines now needs arguing rather than assuming.
What this wave changed in the index
Two new nodes anchor the ingestion story — Supermetal and RisingWave — joining the three agent-memory nodes added earlier this week (Cloudflare Agent Memory, Evermind EverOS, Fastio). Enrichment passes landed on Apache Polaris, Project Nessie, Lakekeeper, Flink CDC, Apache Spark, Delta UniForm (4.3's atomic incremental conversion), Milvus (3.0-beta's zero-copy lake queries), WEKA (NeuralMesh on the NVIDIA AI Data Platform design), VAST Data, Apache Paimon, and — because local-first is quietly becoming an object-storage story — CRDT, where VSCode's migration off Operational Transform marks the moment conflict-free replication became infrastructure.
The index stands at 416 nodes. The calls in this post — the catalog tilt, Kafka optionality, two-engine erosion — are logged with revision conditions on the Calls Ledger; that's where we score ourselves when the dust settles. The through-line of this wave: storage formats stopped being the war. Now the fight is over who gets to sit between you and them — and the answer, increasingly, is: as few layers as possible, and every one of them vending scoped credentials.
Works cited
Footnotes
-
The Frontier Moved Again — and the Floor Compounded — the July 5 field report. ↩
-
GHSA-w76p-3cgp-qfcm — Apache Polaris metadata write bypass — CVE-2026-42812 and the credential-vending CVE cluster, fixed in 1.4.1. ↩
-
Apache Polaris 1.6.0 — structured STS session names, event-listener isolation, strict entity validation; The State of Apache Polaris in July 2026 — TLP graduation and Cloudera adoption context. ↩
-
Best Catalog for Apache Iceberg? (LakeOps) and Iceberg REST Catalog, Nessie & Polaris Compared (PipeCode, June 2026) — the greenfield-caution verdict and the convergence-into-Polaris expectation. ↩
-
Lakekeeper release notes — 0.12 audit events, batch authorization, Generic Table API. ↩
-
Postgres to Iceberg in 13 minutes (The New Stack, April 2026) — normalized single-node TPC-H SF50 benchmark. ↩
-
Supermetal — Announcing the Apache Iceberg target — the WAL→Arrow→Parquet architecture and phase-aware writer. ↩
-
Apache Flink CDC 3.6.0 release announcement — schema evolution, VARIANT type, Oracle source, Hudi sink. ↩
-
How Bytewax Beats Flink in Efficiency, Cost, and Ease of Use — the McKnight Consulting Group benchmark. ↩
-
Announcing General Availability of Real-Time Mode (Databricks) and Breaking the microbatch barrier — RTM GA, architecture, and the Coinbase/DraftKings deployments. ↩ ↩2