Credential Vending in Modern Data Lakes

Distributing static IAM access keys to distributed compute clusters — Spark executors, Trino workers, Flink task managers — is a security liability: keys are long-lived, broadly scoped, and stored in configuration files or environment variables that are difficult to rotate. Modern Iceberg REST catalogs implement credential vending, evaluating fine-grained access policies at the catalog level and issuing temporary, prefix-scoped storage tokens to query engines on a per-request basis. Engineers need to understand how credential vending works, which catalogs support it, and how to migrate from static key distribution.

• Topics: Metadata Management
• Technologies: Apache Polaris, Unity Catalog, AWS Glue Catalog, Hive Metastore
• Standards: Iceberg REST Catalog Spec
• Architectures: Encryption / KMS, Tenant Isolation, Row / Column Security

1. Assess your current credential distribution model. Identify how compute nodes currently authenticate to S3. Common patterns:

   • Static IAM access keys in Spark/Trino configuration files.
   • Instance profiles or IAM roles for EC2/EKS-based compute (better, but still broadly scoped to the role's policy).
   • Service accounts with IRSA (IAM Roles for Service Accounts) on EKS — scoped per pod, but policy management is complex at scale.

2. Understand the REST catalog credential vending flow. When a query engine calls loadTable on an Iceberg REST catalog, the catalog evaluates the caller's identity against its access policy, then returns the table metadata along with temporary S3 credentials (STS tokens) scoped to that table's S3 prefix. The engine uses these credentials for the duration of the query. No static keys are distributed.

   • Credentials are short-lived (typically 15 minutes to 1 hour).
   • Credentials are prefix-scoped — they grant access only to the specific S3 paths that the table occupies.

3. Configure Polaris or Unity for token vending. Apache Polaris implements credential vending per the Iceberg REST Catalog Spec. Unity Catalog provides a similar mechanism through its own API.

   • Polaris: configure an IAM role that the catalog assumes, with permissions to generate STS tokens scoped to each table's S3 prefix.
   • Unity: configure external storage locations with credential passthrough.
   • Both require the catalog service to have IAM permissions to call sts:AssumeRole.

4. Define prefix-scoped access policies. Map your table-to-S3-prefix relationship. Each table should occupy a distinct S3 prefix (e.g., s3://warehouse/db/table/) to enable prefix-scoped credentials. If multiple tables share a prefix, credential scoping is limited to the shared prefix — reducing isolation.

   • Namespace-level policies allow grouping tables with similar access requirements.

5. Test multi-tenant isolation. In multi-tenant environments, credential vending ensures that Tenant A's query engine cannot access Tenant B's S3 data — even if both tenants use the same compute cluster. Test by issuing queries from one tenant's identity and verifying that S3 access to another tenant's prefix is denied.

   • This is the primary security advantage over instance profiles, which grant the same permissions to all processes on a node.

6. Deprecate static key distribution. Once credential vending is operational, remove static keys from configuration files, rotate existing keys, and update operational runbooks. Set up monitoring to detect any compute node that falls back to static key authentication.

   • Implement a grace period where both mechanisms are active, then enforce vending-only access.

• Static IAM keys were the default authentication mechanism for distributed compute on S3 from 2010 through 2020. Key rotation was manual and infrequent.
• Instance profiles (2012) improved security by eliminating static keys for EC2-based compute, but policies were node-scoped, not query-scoped.
• The Iceberg REST Catalog Spec (2022) formalized credential vending as a first-class catalog capability, enabling per-table, per-query credential issuance.
• Apache Polaris and Unity Catalog (both open-sourced 2024) implemented production-grade credential vending, making it accessible without building a custom catalog.
• The trend is toward zero-standing-privileges: compute nodes have no inherent S3 access and receive scoped credentials only when executing authorized queries.