The Catalog Became the Control Plane — for Humans and Agents Alike

A few weeks ago we argued that the catalog is becoming a database — an independent system with its own query language, concurrency model, and storage layout, not a metadata sidecar bolted to the table format. That post was about the catalog's internals getting serious.

This is the companion shift, and it's about the catalog's authority. In the second quarter of 2026, Apache Polaris, Unity Catalog, and Apache Gravitino — three projects that compete on almost everything — all moved in the same direction within a single quarter. The catalog stopped being the place you ask where is the table and became the place that decides who gets to touch it, with what credentials, after which planning step, and — newly — whether the thing asking is a person or an agent.

We have a name for the pattern now: the Catalog-Centric Control Plane. The underlying pain points are ones this index mapped long before the convergence: Vendor Lock-In, Metadata Overhead at Scale, and — the newest and most interesting — the Tool Discovery Governance Gap, the failure mode where AI agents get an ungoverned side door to data that human users never had.

Why the engine can't own governance anymore

The forcing function is simple. In 2022 a lakehouse table was read by one or two engines you controlled. In 2026 the same Apache Iceberg table on object storage is read and written by Apache Spark, Trino, DuckDB, Snowflake, ClickHouse — and a growing fleet of AI agents that nobody provisioned a database login for.

When that many independent clients share one physical copy, governance cannot live in any one engine. If Spark enforces row-level security but the agent reads the Parquet files directly, the policy is decorative. The only place a rule can be enforced for everyone is the layer they all have to go through to find and open the table: the catalog. So the responsibilities migrate inward — authorization, short-lived credential vending, federation across clouds, even query planning. The catalog becomes the control plane because it's the last place left that sees every request.

Signal 1 — Polaris 1.5: governance becomes pluggable, and the UC gap closes

Apache Polaris 1.5.0 (May 18, 2026) is the clearest statement of intent. It refactored authorization into a fully pluggable Authorizer SPI — request-scoped AuthorizationRequest / AuthorizationDecision interfaces that cleanly separate the access decision from Polaris's internal principal resolution.1 In plain terms: Polaris stopped hard-coding how it answers "is this allowed?" and made it a swappable component.

The immediate payoff is Apache Ranger support (Beta)1 — and that's the strategically loaded part. "Unity Catalog has enterprise governance, Polaris doesn't" had been the single biggest reason organizations reached for the proprietary option over the open Iceberg REST reference implementation. With Apache Ranger integration, that reason is evaporating.

1.5 also pushed the control-plane envelope on two more axes: BigQuery Metastore federation (Polaris serves a unified Iceberg REST interface over tables that physically live in BigQuery Metastore) and credential vending extended to the Generic Table API via the new Polaris-Generic-Table-Access-Delegation header — so short-lived, prefix-scoped storage credentials now cover non-Iceberg datasets, not just Iceberg tables.1 Federation and credential brokering are control-plane jobs, not metadata-lookup jobs.

Signal 2 — Gravitino: planning moves into the catalog

Apache Gravitino 1.2.0 (March 13, 2026) attacks the control plane from the engine side. Its Iceberg REST server now lets query engines like DuckDB and Spark offload scan planning to the catalog, backed by a scan-planning cache.2

This is subtle but important. Scan planning — figuring out which data files a query actually needs to touch — has always been the engine's job. Moving it into the catalog means the catalog now mediates not just whether you can read a table but how the read is shaped. Once planning, credential vending, and policy all sit in one place, that place is unambiguously the control plane, and the Iceberg REST Catalog Spec is the substrate it runs on.

Signal 3 — the control plane turns to face the agents

The newest axis, and the reason this is more than a re-run of the 2019 "central metastore" debate, is who is asking. The same catalogs that are absorbing human governance are simultaneously becoming the access layer for AI agents:

  • Gravitino ships an MCP server plus a Model Catalog, exposing governed metadata to AI tools — both introduced in the 1.1.0 "AI-native metadata management platform" release (December 16, 2025).3
  • Databricks' managed MCP servers expose Unity Catalog tables, functions, and Vector Search indexes to agents natively.4
  • ClickHouse, at Open House 2026 (May 27), launched Claude-powered ClickHouse Agents into public beta and shipped clickhousectl with an agent-skills installer — making the stack drivable by the agentic development loop end-to-end.56

Connect the dots and the Tool Discovery Governance Gap gets a real answer. When an agent reaches data through the catalog's managed MCP server, it inherits the same authorization boundary a human principal would — instead of being handed a broad service token and trusted to behave. The catalog becomes the one place where "scope the agent's access" and "scope the analyst's access" are the same operation.

Even Microsoft's OneLake — Azure-side, not S3 — joined the open lakehouse through exactly this seam: it exposes an Iceberg REST Catalog API so external engines query Fabric tables via standard connection strings, with Snowflake↔OneLake bidirectional interop now GA.7 The catalog API, not the storage API, is how a vendor opts into the ecosystem in 2026.

What this means if you're building on S3

Three practical consequences:

  1. Pick the catalog before the engine. The catalog is now where lock-in and governance are decided. Choosing Polaris vs Unity Catalog vs Gravitino is a more consequential decision than choosing Spark vs Trino, because the catalog is the layer everything else — including your agents — has to pass through.8
  2. Route agents through the managed MCP server, not around it. The governance guarantee only holds if the agent's path is the catalog's path. An agent with raw object-store credentials bypasses every control the catalog enforces.
  3. Stop vending long-lived keys. Credential vending — short-lived, prefix-scoped credentials minted by the catalog at table-load time — is now table stakes across Polaris, Unity Catalog, and the Iceberg REST spec itself. Long-lived S3 keys handed to engines are the legacy pattern this replaces.

In 2024 the catalog remembered where the tables lived. In 2026 it decides who — and what — gets to read them. The most interesting database in the lakehouse is also now its security perimeter.

Footnotes

  1. Apache Polaris 1.5 release — pluggable Authorizer SPI, Apache Ranger (Beta), BigQuery Metastore federation, Generic-Table credential vending — Snowflake Engineering Blog; release artifacts at Apache Polaris 1.5.0 downloads. ↩ ↩2 ↩3

  2. Apache Gravitino 1.2.0 — scan-planning offload to the IRC server with a scan-planning cache — Gravitino 1.2.0 release notes. ↩

  3. Apache Gravitino 1.1.0 — Model Catalog + MCP server, "An AI-native metadata management platform" — Gravitino 1.1.0 release notes. ↩

  4. Unity Catalog managed MCP servers expose tables, functions, and Vector Search indexes to agents; full Iceberg GA over the REST API — Databricks — Unity Catalog and the next era of Apache Iceberg; Expanded interoperability with Unity Catalog Open APIs. ↩

  5. ClickHouse Agents (Claude-powered, public beta) at Open House 2026, May 27, 2026 — ClickHouse — Open House 2026 Day 1; ClickHouse tops $250M ARR + Claude-powered Agents. ↩

  6. clickhousectl (beta, April 9, 2026) — JSON-emitting CLI + agent-skills installer for 15+ coding agents — Futurum — ClickHouse Builds a CLI to Make its Databases Agent-Native. ↩

  7. Microsoft OneLake Iceberg REST Catalog API + Snowflake interoperability GA — Microsoft Fabric — access your Fabric tables in Apache Iceberg format; OneLake + Snowflake interoperability is now GA. ↩

  8. Comparative framing of Polaris vs Unity Catalog vs cloud REST catalogs as control-plane choices — DataLakehouseHub — Choosing the Right Iceberg Control Plane (May 2026). ↩