Technology

Hive Metastore

The original metadata catalog service from the Apache Hive project that stores table schemas, partition mappings, and storage locations for data on S3 and HDFS. Commonly abbreviated as HMS.

9 connections3 resources1 post

Summary

What it is

The original metadata catalog service from the Apache Hive project that stores table schemas, partition mappings, and storage locations for data on S3 and HDFS. Commonly abbreviated as HMS.

Where it fits

Hive Metastore is the legacy but still widely deployed catalog underpinning Spark, Trino, Presto, and Flink workloads against S3 data. It predates dedicated Iceberg catalogs and remains the default metastore for many on-premise and hybrid lakehouse deployments.

Misconceptions / Traps

HMS was designed for Hive partition-based tables. Its data model is a poor fit for Iceberg's snapshot-based metadata, which is why dedicated Iceberg catalogs (REST, Nessie, Glue) are preferred for new deployments.
Running HMS requires a backing relational database (MySQL, PostgreSQL). That database becomes a single point of failure and a scaling bottleneck for metadata operations.
HMS is not a governance tool. It stores structural metadata but has no built-in access control, lineage tracking, or data quality features.

Key Connections

scoped_to Metadata Management — the original Hadoop-era catalog
enables Apache Spark, Trino, Apache Flink — query engines that read HMS metadata
alternative_to AWS Glue Catalog, Apache Polaris — older alternative to managed catalogs
constrained_by Metadata Overhead at Scale — HMS database becomes a bottleneck at large scale

Definition

What it is

An open-source metadata service originally built for Apache Hive that stores table schemas, partition locations, and statistics for data stored on HDFS or S3. The longest-standing catalog in the Hadoop ecosystem.

Why it exists

Before table formats, Hive Metastore was the primary way to impose table structure on files stored in distributed storage. It remains widely deployed as the default catalog for Spark, Trino, and Flink workloads reading from S3.

Primary use cases

Legacy catalog for Spark and Trino workloads, Iceberg catalog backend (HiveCatalog), schema registry for Hive-style partitioned tables on S3.

Recent developments

Latest signals

Hive 4.2.0 released (Nov 23, 2025) — Iceberg REST Catalog client baked in. Per the Apache Hive downloads page, the newest release raises the minimum runtime to JDK 21 and adds enhanced Iceberg support: deletion vectors, auto compaction, ViewCatalog, column defaults, an Iceberg REST Catalog client, Z-ordering, and variant type. Notably, the project itself now ships tooling to talk to the REST catalog spec that is displacing HMS.
Standalone Metastore split out as its own component (Hive 4.1.0). Per Apache Hive downloads, the 4.1.0 release (Jul 31, 2025) made the Hive Metastore available as a standalone component in both binary and Docker image formats, decoupled from the full Hive distribution — the packaging most Spark/Trino/Flink deployments actually want.
Databricks deprecates Hive Metastore for new workspaces — September 30, 2026 cutoff, all three clouds. Per the Databricks UC-only migration docs, starting September 30, 2026 new Databricks workspaces on AWS, Azure, and GCP will be provisioned without Hive metastore access, in favor of Unity Catalog. Existing workspaces are not auto-migrated, but the direction is unambiguous: HMS is the legacy path.
Iceberg ecosystem trend: REST Catalog displacing Hive Metastore as default. Per RisingWave's Iceberg 2026 analysis, Hive Metastore is consistently characterized as "complex and stateful" relative to the Iceberg REST Catalog spec, which has become the standard. Engines support HMS, Glue, and REST concurrently — but new architectural decisions in 2026 uniformly choose REST Catalog over HMS. For teams on HMS, this is a "stable but no-new-investment" signal, not an immediate-migration mandate.
Practitioner survey (January 2026): Hive Metastore adoption sits at 18%, trailing AWS Glue (39%), Nessie (29%), S3 Tables (25%), and Polaris (21%). A concrete adoption number, not just roadmap framing — quantifies how far HMS has fallen behind newer catalog options in real deployments. Per Datamy — Data Warehouse/Lake/Lakehouse: How to Choose for Business.
The performance case against HMS, in detail: Thrift RPC scaling limits, RDBMS strain at large partition counts, and poor object-storage-specific optimization. Elaborates the mechanism behind the Object Listing Performance pain point already listed on this node — why HMS specifically struggles on S3-backed workloads. Per lakeFS — Hive Metastore's Dilemma: Performance Limits (March 2026).
Still drawing active third-party integration work despite legacy status. The SQE open-source catalog compatibility matrix added live integration tests against standalone Hive Metastore 4.1.0 over Thrift, scoring 158/189 (83.6%); a DuckDB extension for querying HMS directly as a catalog also remains under active development as of mid-2026. Per SQE — Next Steps and DuckDB Extension Radar.
The cutoff's sharpest edge is cluster access modes, not the metastore. After migration, Unity Catalog imposes Standard (shared) or Dedicated (single-user) access modes — and Standard prohibits RDD APIs, SparkContext, ML runtimes, and complex UDFs, leaving heavily customized multi-tenant ETL/ML pipelines a choice between DataFrame-API rewrites and losing shared-cluster economics on Dedicated mode. No canonical Databricks guidance yet exists for these code-level incompatibilities. Per Databricks Community — HMS to UC migration playbook. Sources: Apache Hive downloads (4.2.0 / 4.1.0 release notes) · Migrate your account to UC-only workspaces (Databricks docs) · Apache Iceberg in 2026 (RisingWave)