Technology

DataHub

An open-source metadata platform originally developed at LinkedIn that provides data discovery, lineage tracking, governance, and observability across data lake and lakehouse environments.

9 connections3 resources1 post

Summary

What it is

An open-source metadata platform originally developed at LinkedIn that provides data discovery, lineage tracking, governance, and observability across data lake and lakehouse environments.

Where it fits

DataHub serves the same governance layer as OpenMetadata, providing search-driven data discovery over S3-based assets. It differentiates with a stream-based metadata architecture (built on Kafka) and a GraphQL API for programmatic metadata access.

Misconceptions / Traps

DataHub's metadata ingestion is source-pull, not real-time push. There is a delay between changes in source systems and their appearance in DataHub's catalog.
DataHub's Kafka-based metadata store adds operational complexity. Running DataHub requires Kafka, Elasticsearch, MySQL, and a graph database (Neo4j or relational).
Lineage in DataHub depends on source system instrumentation. If a Spark job does not emit OpenLineage events, DataHub will not automatically detect the lineage.

Key Connections

scoped_to Metadata Management — metadata discovery and governance platform
depends_on Kafka Tiered Storage — uses Kafka for metadata event streaming
alternative_to OpenMetadata, Apache Atlas — competing metadata platforms
enables Audit Trails — lineage and change tracking for compliance

Definition

What it is

An open-source metadata platform originally developed at LinkedIn that provides data discovery, lineage, governance, and observability for S3-based data lakes. Supports automated metadata ingestion from Iceberg, Delta, Hive, and other S3-centric sources.

Why it exists

Large organizations with S3-based data lakes need to know what data exists, how it flows between systems, and who is responsible for it. DataHub provides a metadata graph with real-time ingestion, search, and governance workflows.

Primary use cases

Enterprise data cataloging over S3 lakehouses, automated lineage from Spark/Airflow pipelines, data governance and compliance workflows.

Recent developments

Latest signals

Two 2026 CVEs — both fixed; patch floor is 1.5.0.3. CVE-2026-25644 (February): LDAP ingestion TLS-downgrade MITM, fixed in 1.3.1.8. CVE-2026-44501 (May, CVSS 7.1): unsafe Java deserialization of the REDIRECT_URL cookie in the OIDC callback — an RCE-class flaw — fixed in 1.5.0.3. Self-hosted deployments below those floors should upgrade before wiring the catalog to agents. Per NVD — CVE-2026-25644 and NVD — CVE-2026-44501.
v1.6.0 internals: async Kafka offset commits deliver up to 25× ingestion throughput. The Play 3 + Apache Pekko frontend migration (V1 UI deleted) rides alongside GMS on Spring Boot 4 + Java 21/25 — custom GMS plugins must recompile — and the async-commit ingestion path (with an event-redelivery caveat under failure). Acryl's commercial layer (Acryl Observe: freshness/volume/column assertions with lineage-driven alerting; ~$30M raised; Snowflake Premier-tier partner 2026) monetizes the same metadata plane. Per DataHub releases (docs) and Acryl Observe.
Latest release: v1.7.0 (August 2026), succeeding v1.6.0 (May 21, 2026). Per datahub-project/datahub releases.
DataHub Cloud v1 launched as a "context platform" for analytics agents (May 28, 2026). The release repositions DataHub as a context layer between analytics agents — Databricks Genie, Snowflake Intelligence — and enterprise data, feeding them unified metadata, semantic definitions (dbt, Power BI), and institutional knowledge (Notion, Confluence) so agents generate correct SQL; DataHub reports pushing agent accuracy beyond 90%. It is the same agent-context pattern as the broader MCP wave, applied to the metadata catalog. Per DataHub Launches Breakthrough Release for Analytics Agents.
Open-source DataHub v1.6.0 (May 21, 2026) went V2-UI-only. The OSS release removed the legacy V1 UI entirely and moved datahub-frontend onto Play 3 + Apache Pekko for improved security and maintainability. Per DataHub Releases (docs.datahub.com).
#1 open-source AI data catalog framing — 80+ production-grade connectors, MCP support. Per the DataHub Project GitHub organization, DataHub positions itself as "The Context Platform for your Data and AI Stack" with 80+ production-grade connectors, real-time streaming metadata updates, AI-readiness with MCP support, and LLM-friendly metadata exposure. The MCP integration is the same architectural pattern that Snowflake, Databricks, and dlt are converging on: catalog metadata becomes a primary surface for AI-assisted analytics rather than a backend governance layer.
Active ecosystem positioning vs OpenMetadata. Per the 16 Best Data Catalog Tools in 2026 buyer's guide, DataHub is recommended for organizations that want "most active open-source community, API-first metadata ingestion, Python/Java engineering resources" — 11,600+ GitHub stars and a three-year head start over OpenMetadata on community size. The 2026 framing positions DataHub as the choice when engineering resources are available; managed-product alternatives (Atlan, Collibra) cover the no-engineering-team segment.
The "trusted context" gap is the quantified thesis (State of Context Management Report, March 2026). Acryl's report puts numbers on why catalogs pivoted to agent-context: 90% of organizations claim to be "AI-ready," yet 87% name a lack of trusted data context as their primary blocker to putting agents in production, and 61% frequently delay AI initiatives over untrusted data. That gap is the market DataHub Cloud v1 targets. To let orgs close it without forking core, DataHub also shipped Micro Frontends — runtime injection of custom, organization-specific workflows into the DataHub UI. Per DataHub Releases State of Context Management Report 2026 and DataHub March 2026 Town Hall: Agents, Apps & More.
Open-source DataHub Analytics Agent shipped under Apache 2.0, built on MCP + Agent Context Kit. Announced at the April 2026 town hall with Pinterest cited as a production user running DataHub as the context substrate for its AI agents; docs confirm the agent answers data questions in plain English and ships an MCP server for direct integration with Cursor, Claude Desktop, and Cline. Per DataHub April 2026 Town Hall and DataHub Docs Overview.
DataHub's open-source roots trace to LinkedIn's GDPR compliance work, and the customer roster now spans Netflix, Apple, JPMorgan Chase, GSK, Visa, and Figma for cross-asset discovery (data, ML, and software). Per About DataHub. (tertiary)
DataHub Cloud carries a 99.5% uptime SLA with SOC 2 Type II and HIPAA compliance, positioning the managed tier for regulated enterprise buyers alongside the Ask DataHub conversational assistant and ML-based anomaly detection. Per DataHub Cloud Reviews — Gartner Peer Insights. (tertiary)