Standard

Iceberg REST Metrics API

A 2026 formalization of the Iceberg REST Catalog (IRC) specification that adds a dedicated telemetry-reporting endpoint — `POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics` — through which compute engines (Spark, Trino, Flink, and other IRC-compliant clients) push query-level execution telemetry back to the catalog after a scan or commit completes. Reports are typed **ScanReport** (rows read, bytes processed, filter/query duration, snapshot read) and **CommitReport** (commit duration, added/removed files, operation type) payloads, each carrying a shared `trace-id` field so a single logical query's scan and any resulting commit can be correlated at the catalog layer, even across engine boundaries.

6 connections

Definition

What it is

A 2026 formalization of the Iceberg REST Catalog (IRC) specification that adds a dedicated telemetry-reporting endpoint — `POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics` — through which compute engines (Spark, Trino, Flink, and other IRC-compliant clients) push query-level execution telemetry back to the catalog after a scan or commit completes. Reports are typed **ScanReport** (rows read, bytes processed, filter/query duration, snapshot read) and **CommitReport** (commit duration, added/removed files, operation type) payloads, each carrying a shared `trace-id` field so a single logical query's scan and any resulting commit can be correlated at the catalog layer, even across engine boundaries.

Why it exists

Before this endpoint, query-level cost, performance, and access telemetry lived wherever the compute engine chose to emit it — Spark UI, Trino's event-listener framework, Flink metrics — none of which the catalog could see or tie back to the RBAC/credential-vending decision it had already made to grant that engine table access. The catalog knew *who was authorized*; it had no record of *what they actually did*. The /metrics endpoint closes that loop: because every table load and credential vend already flows through the catalog, having engines report execution telemetry back through the same channel turns the catalog into the single place where access grants and actual usage can be joined — without instrumenting the compute layer at all.

Primary use cases

Cost allocation per table/namespace/principal (bytes-processed billing without engine-side cost-tracking plugins); catalog-layer audit trails that tie a credential grant to the query that used it; security forensics (reconstructing what a compromised identity actually scanned, not just what it was authorized to scan); cross-engine query observability (one telemetry stream regardless of whether Spark, Trino, or Flink executed the query); SLA/performance monitoring without deploying per-engine listeners.

Recent developments

Latest signals
  • Formalized in the IRC OpenAPI spec as a first-class endpoint. POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics accepts ScanReport and CommitReport payloads carrying a shared trace-id, so a scan and any commit it triggers can be joined into one telemetry record at the catalog layer — a substantive evolution of the IRC specification, not an incremental addition. Per apache/iceberg issue #15521.
  • Apache Polaris and Lakekeeper are the reference implementers building the metrics-persistence layer. Both catalogs already sit on the request path for every table load and credential vend; adding metrics persistence there — rather than in the compute engine — lets each fuse its existing authorization decision (Polaris's RBAC, Lakekeeper's embedded Cedar/OPA policies) with a record of what was actually read or written per request. Per Lakekeeper Management API docs and apache/polaris issue #3337.
  • Bundled with a broader "catalog as DBMS" OpenAPI expansion. The same 2026 spec-evolution push tracked in apache/polaris #3337, apache/polaris #5054, and apache/iceberg #15521 also proposes soft-delete lifecycles, an undrop API, and standardized label structures for cost attribution — the metrics endpoint is the observability half of a maturation arc that includes lifecycle and cost-tagging primitives, moving the REST Catalog spec from "metadata lookup contract" toward a comprehensive database-management-system surface. Per apache/polaris issue #3337, apache/polaris issue #5054, and apache/iceberg issue #15521.

Connections 6

Outbound 6