Standard

Iceberg REST Catalog Spec

An open REST API specification for Apache Iceberg catalog operations — namespace/table listing, metadata load, commit, snapshot management — enabling multi-engine interoperability through a standardized HTTP-based catalog interface. Extended in practice with **credential vending**, where the catalog mints prefix-scoped, short-lived S3 credentials at table-load time.

17 connections6 resources4 posts

Summary

What it is

An open REST API specification for Apache Iceberg catalog operations — namespace/table listing, metadata load, commit, snapshot management — enabling multi-engine interoperability through a standardized HTTP-based catalog interface. Extended in practice with **credential vending**, where the catalog mints prefix-scoped, short-lived S3 credentials at table-load time.

Where it fits

The REST Catalog Spec solves the catalog fragmentation problem in the Iceberg ecosystem. Instead of every engine needing native support for Hive Metastore, Glue, Nessie, etc., any catalog that implements the REST spec becomes accessible to all REST-capable engines. This is also the wire that lets a local-first engine like DuckDB attach directly to an Amazon S3 Tables bucket (`ATTACH '<arn>' AS cat (TYPE iceberg, ENDPOINT_TYPE s3_tables)`) or metadata-clone via `iceberg_to_ducklake(...)` — bypassing heavy distributed compute for interactive querying of multi-terabyte remote tables.

Misconceptions / Traps
  • The REST Catalog Spec defines the API contract, not the catalog implementation. Performance, consistency, and feature completeness depend on the catalog server behind the API.
  • Not all Iceberg catalog operations may be supported by every REST catalog implementation. Check compatibility for advanced features like branching, tagging, and view support.
  • Credential vending is not part of the base spec — it's a widely-adopted extension (Apache Polaris, Unity Catalog, S3 Tables). Check whether your client understands the vended-credential response shape before assuming it "just works."
Key Connections
  • scoped_to Iceberg Table Spec, Table Formats — standardizes catalog access for Iceberg
  • used_by DuckDB — the direct-attach path for local-first analytics over S3 Tables
  • solves Vendor Lock-In — engine-agnostic catalog access
  • solves Metadata Overhead at Scale — enables centralized catalog management

Definition

What it is

An open specification defining a RESTful HTTP API for Iceberg catalog operations — listing namespaces and tables, loading table metadata, committing updates, and managing snapshots — independent of any specific catalog backend. Extended in practice with **credential vending**, where the catalog mints short-lived, prefix-scoped S3 credentials at table-load time so clients never hold broad long-lived keys.

Why it exists

Iceberg catalogs were historically tied to specific implementations (Hive Metastore, AWS Glue, Nessie). The REST Catalog Spec decouples catalog clients from catalog backends, enabling multi-engine, multi-language interoperability through a universal HTTP interface.

Primary use cases

Multi-engine Iceberg catalog access, vendor-neutral catalog interoperability, cloud-managed Iceberg catalogs, cross-language catalog clients, local-first analytics (DuckDB `ATTACH '<s3_tables_arn>' AS cat (TYPE iceberg, ENDPOINT_TYPE s3_tables)`), DuckLake metadata-only cloning (`CALL iceberg_to_ducklake(...)`) for interactive querying of multi-terabyte remote tables.

Recent developments

Latest signals

  • Vendors. Official vendor page listing 20+ vendors: Confluent, Crunchy Data, Databricks, dltHub, Firebolt, Fivetran, Google Cloud BigLake, IBM watsonx.data, IOMETE, Microsoft OneLake, Oracle, Snowflake, StarRocks, and more. Per iceberg.apache.org.

  • May 7, 2026: Support for Apache Iceberg™ version 3 (General availability). Snowflake GA support for Iceberg v3. New types: geography, geometry, nanosecond timestamp, variant. Features: default values, deletion vectors, row lineage. Horizon Iceberg REST Catalog API for external engine reads. Per Snowflake Engineering Blog (2026-05-07).

  • [Bug report] Trino creates a table with format_version = 3 for a remote IRC, but the resulting table ends up with format_version = 2. Gravitino Iceberg REST server bug where Trino requests format_version=3 but table is created as v2. Involves credential vending and S3 token generation. Per GitHub (apache/gravitino) (2026-05-07).

  • The REST Catalog is becoming a control-plane substrate, not just a metadata lookup. Two 2026 moves push the spec past "list namespaces, load metadata." First, scan-planning offload: Apache Gravitino's IRC server now performs Iceberg scan planning on behalf of engines like DuckDB and Spark (with a scan-planning cache), shifting planning from client to catalog. Second, the IRC endpoint is now the seam vendors use to expose their own table estates to the open ecosystem — Microsoft OneLake exposes an Iceberg REST Catalog API so external engines (Snowflake, Dremio, Trino) query Microsoft Fabric tables via standard IRC connection strings, with Snowflake↔OneLake bidirectional interoperability now GA. The catalog API is quietly absorbing planning, credential vending, and cross-vendor federation — the control surface of the lakehouse. Per Gravitino 1.2.0 release notes, Microsoft Fabric — Iceberg support in OneLake, and Microsoft — OneLake and Snowflake interoperability is now GA.

  • Apache Polaris graduated to Apache Software Foundation Top-Level Project (February 2026). Polaris — the open-source, vendor-neutral REST Catalog implementation with RBAC, multi-tenant namespaces, credential vending, and remote signing — moved out of incubation; Snowflake's Horizon Catalog is built on the same open-source backbone as its managed service, not a separate edition. Per javatask.dev — Iceberg for OT, Part 4: Catalogs, Security, and Interoperability in 2026 and Snowflake Engineering Blog — Apache Polaris: The End of Data Vendor Lock-In.

  • CVE-2026-34214: Trino's Iceberg REST Catalog connector exposed both static and vended credentials via query JSON to any user holding SQL write privilege — the temporary, vended credentials the spec's credential-vending pattern is meant to protect were affected too, not just static keys. Fixed in Trino 480. Per GitLab Advisory Database — CVE-2026-34214.

  • Iceberg v4 roadmap unveiled at Iceberg Summit 2026, including new REST catalog capabilities-header work for fleet coordination across engines. As of July 2026, relative-path support and content-stats work are decided; single-file commits and an adaptive metadata tree remain debated — timeline isn't locked but direction is. Per The State of Apache Iceberg v4 in July 2026.

  • iceberg-rust's partition-spec serialization isn't spec-compliant — it writes a bare JSON array instead of the required object form, breaking interoperability with iceberg-java, iceberg-cpp, and pyiceberg on fast_append commits against tables with non-Rust snapshots. A concrete crack in the spec's cross-language interoperability promise. Per GitHub (apache/iceberg-rust) #2365.

  • The spec's newest move: a formal /metrics endpoint for query-level telemetry. POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics lets compute engines (Spark, Trino, Flink) push ScanReport/CommitReport payloads — rows read, bytes processed, query duration, linked by a trace-id — back to the catalog, turning IRC from a metadata-lookup contract into a telemetry-collection one as well. Full writeup: Iceberg REST Metrics API. Per apache/iceberg issue #15521. Sources: iceberg.apache.org · Snowflake Engineering Blog · GitHub (apache/gravitino) · Gravitino 1.2.0 release notes · Microsoft Fabric — Iceberg support in OneLake · Microsoft — OneLake + Snowflake interoperability GA

Connections17

Outbound5
scoped_to2
Table FormatsLakehouse
enables1
Lakehouse Architecture
used_by1
DuckDB
solves1
Vendor Lock-In
Inbound12
implements10
Technology9
Amazon S3 TableslakeFSApache PolarisApache GravitinoUnity CatalogLakekeeperAWS Glue CatalogProject NessieMicrosoft OneLake
Architecture1
Catalog-Centric Control Plane
competes_with1
TreeCat
depends_on1
Iceberg REST Metrics API

Resources6

SpecHigh
github.com/apache/iceberg/blob/main/open-api/rest-catalog-op...

OpenAPI specification for the Iceberg REST Catalog, the standard API for multi-engine catalog interoperability.

DocsHigh
iceberg.apache.org/

Iceberg catalog concepts documentation explaining catalog types, REST catalog design, and configuration.

BlogMedium
iceberg.apache.org/docs/latest/

Introduction to the Iceberg REST Catalog from Tabular (founded by Iceberg creators) explaining design motivations and architecture.

DocsHigh
duckdb.org/docs/current/core_extensions/iceberg/amazon_s3_ta...

DuckDB Iceberg extension documentation showing the `ATTACH ... TYPE iceberg ENDPOINT_TYPE s3_tables` pattern for local-first analytics over remote S3 Tables buckets.

DocsHigh
duckdb.org/docs/current/core_extensions/iceberg/iceberg_rest...

Reference for DuckDB's REST Catalog support and the `iceberg_to_ducklake` metadata-only clone path for interactive querying of multi-terabyte remote tables.

BlogHigh
aws.amazon.com/blogs/storage/streamlining-access-to-tabular-...

AWS walkthrough of DuckDB → S3 Tables via REST Catalog credential vending, including predicate pushdown and partition pruning behavior.

Featured in

2026-08-12When the Paper Trail Stopped Matching the SoftwareTwo real concurrency bugs surfaced in Apache Iceberg's own issue tracker. A year-old governance milestone sat uncorrected on a catalog page. A vector database's supposed major-version jump turned out to be a misread dev-build tag. This index runs on citations — so a week spent checking hundreds of them against primary sources is itself the story.2026-07-15Kafka Became Optional — the Pipeline Rebuilt Itself and the Catalog Race TiltedTen days after our last field report, three structural stories crystallized. The catalog race tilted hard on a governance decision: Polaris graduated, shipped a monthly release train, and won Cloudera — while Nessie's Git semantics look set to be absorbed rather than adopted. The ingestion pipeline collapsed from four systems to one: a Rust process that snapshots Postgres into Iceberg in 13 minutes, against Flink's 90. And Spark's Real-Time Mode went GA, closing the batch/streaming split that justified running two engines. The common thread: every intermediate layer is being asked to justify its existence, and most can't.2026-06-10The Catalog Became the Control Plane — for Humans and Agents AlikeWithin one quarter, Apache Polaris, Unity Catalog, and Apache Gravitino all converged on the same role: the catalog is no longer where you look up where a table lives — it's where access is granted, credentials are minted, query planning happens, and AI agents are let in or kept out. The companion shift to 'the catalog is becoming a database': the catalog is also becoming the control plane.2026-04-16When S3 Became a File System: The April 2026 Evolution and What It Means for Local-First BuildersAWS shipped S3 Files on April 7, 2026. NVIDIA productized RDMA for S3 five months earlier. Both moves validated architectural patterns the local-first ecosystem had been proving for years — and both collapse gaps that this index has been mapping since day one.