OpenLineage | LLMS3

Summary

What it is

An open standard that defines a common JSON schema for capturing data lineage events — what datasets were consumed, what was produced, and how transformations connected them.

Where it fits

OpenLineage is the missing observability layer for S3 lakehouses. As pipelines span Spark, Airflow, Flink, and dbt across multiple S3-backed tables, OpenLineage provides the standard format for stitching lineage together into a complete graph, regardless of which orchestrator runs the job.

Misconceptions / Traps

OpenLineage is a standard, not a product. It has no UI — you need a backend like Marquez or Datakin to store and visualize the lineage events.
Integration quality varies by tool. Some integrations (Spark) are mature; others (Flink) are still developing.

Key Connections

enables Marquez — the reference implementation that stores and visualizes OpenLineage events
scoped_to Lakehouse, S3 — lineage tracking for S3 lakehouse pipelines

Definition

What it is

An open standard for data lineage collection that defines a common JSON schema for capturing metadata about data pipeline runs — what datasets were consumed, what was produced, and what transformations occurred.

Why it exists

Data lineage information was historically locked inside individual orchestration tools (Airflow, Spark, dbt). OpenLineage provides a vendor-neutral, open standard so that lineage events from any tool can be collected, correlated, and queried in a consistent format.

Primary use cases

Cross-tool data lineage tracking for S3 lakehouse pipelines, regulatory compliance auditing, pipeline impact analysis and debugging.

Recent developments

Latest signals

Platform-native adoption wave: Snowflake ships native OpenLineage support (Preview); IBM adopts it across watsonx. Snowflake emitting OpenLineage events natively (January 2026, Preview) and IBM standardizing on it across the watsonx platform move the spec from "instrumentation you bolt on" to "telemetry the platform emits" — the consumers-vs-producers pattern maturing into producers-by-default. Per IBM Docs — OpenLineage integration and Atlan — Snowflake data lineage best practices (aggregator).
Apache Airflow OpenLineage provider 2.19.0 adds a unified emission_policy config. One knob for when/what lineage events are emitted across operators — a sign the Airflow integration has moved past bolt-on and into operational-tuning territory. Per Airflow OpenLineage provider changelog.
Regulated-industry pull: market research reports 68% of Databricks financial-services customers have enabled OpenLineage metadata emission. Market-research-sourced figure (treat the precision loosely) — but the direction matches the EU-AI-Act-driven demand for lineage bridging ML and data pipelines. Per Dataintelo — OpenLineage in finance market report.
LF AI & Data Foundation Graduate project. OpenLineage achieved Graduate-tier status in the LF AI & Data Foundation — the highest open-source maturity signal in the foundation's tiering. Per GitHub — OpenLineage/OpenLineage.
Positioned as "OpenTelemetry for data pipelines." OpenLineage explicitly frames itself as the data-pipeline analog of OpenTelemetry: an API to collect lineage events, agnostic to the backend, aimed at being embedded in every data-processing engine. Per OpenLineage Blog — How OpenLineage takes inspiration from OpenTelemetry.
Trino added native OpenLineage integration alongside OpenTelemetry. Trino's adoption is the bellwether — when the leading lakehouse query engine ships OL + OTel side-by-side, the "two standards, complementary models" framing wins the architectural debate vs the "one standard for everything" camp. Per Improving — Effective Data Lineage Strategies for Real-Time Systems.
First-class integrations: Airflow, Spark, dbt, Flink. The four most-deployed data-pipeline tools all ship native OpenLineage emitters — collection coverage is no longer the gap. The 2026 work is on consumption + visualization (Marquez, DataHub, Atlan, etc.). Per GitHub — OpenLineage/OpenLineage.
USENIX SREcon EMEA 2025: OpenLineage as foundational layer for data reliability. Recognized at the SRE conference circuit — Obuchowski's talk frames OpenLineage as the load-bearing instrumentation layer for "data SRE" the same way OTel grounded service SRE. Per USENIX SREcon EMEA 25 — Cross-Platform Data Lineage with OpenLineage.
OTel spec issue #3447 explores modeling lineage in OTel directly. Active discussion in the OpenTelemetry specification repo on whether to model data lineage as native OTel signals. Outcome will shape whether OL stays a sister project or eventually folds into OTel. Per open-telemetry/opentelemetry-specification Issue #3447.
OpenLineage 1.47.0 adds a standardized batch API endpoint for submitting multiple lineage events in a single request. A concrete spec/API capability addition beyond the existing single-event emission model. Per GitHub — OpenLineage/OpenLineage releases.
Dagster ships native OpenLineage emission, joining Airflow, Spark, dbt, and Flink as first-class producer integrations — the fifth major orchestration/compute engine with a built-in emitter rather than a bolt-on plugin. Per Socratopia — Data Observability and Lineage.
Compatibility maturity report (Q1 2026): 30+ vendor conversations completed, but 141 open compatibility issues remain and three concrete gaps are identified — a Financial Services Working Group, standardized consumption events, and stronger integration testing. Per Bearingnode — OpenLineage Compatibility Update: Q1 2026.
IBM specifically contributing spec enhancements for unified lineage across structured and unstructured data, aimed at enabling explainable AI — a more specific framing than general watsonx adoption, since it targets the spec's metadata model rather than just consuming it. Per IBM — OpenLineage for a unified lineage view.
Consumer/visualization ecosystem now explicitly spans Marquez, Atlan, DataHub Cloud, OpenMetadata, and Microsoft Fabric, per a lineage-tooling survey — broader than the "Marquez, DataHub, Atlan, etc." shorthand already on file. Per Data Stack Index — Marquez.
New RFC: AgentRunFacet extends OpenLineage to track autonomous AI agent actions, not just pipeline runs. The proposal — open as a GitHub Discussion, not yet merged into the spec — adds a facet capturing agent-specific run metadata on top of OpenLineage's existing event-emission model for Airflow/Spark/dbt/Flink/Dagster runs, positioning OpenLineage as lineage infrastructure for agentic AI actions as well as ETL pipelines. Per OpenLineage/OpenLineage Discussion #4407 — AgentRunFacet.
OpenLineage event emission has been directly implicated in a production-corrupting Iceberg concurrency bug. Apache Iceberg issue #17338 ("Race Condition in CachingCatalog") traces to background-thread loadTable() calls triggered automatically by OpenLineage event emission — these calls can cause active writer threads to read stale cached table references, corrupting data during concurrent writes. A related bug (#11664) causes Iceberg to silently swap in a default LoggingMetricsReporter instead of a user-configured one, dropping CommitReport lineage events while ScanReport events keep flowing normally. Full technical detail lives on the Apache Iceberg node — flagged here because both bugs sit on the lineage-emission path this spec defines, and the source report ranks this its #6 top action item for lakehouse governance stacks. Per apache/iceberg Issue #17338 and apache/iceberg Issue #11664.
Latest release: 1.52.0 (July 23, 2026). Per OpenLineage 1.52.0 release notes. Sources: GitHub — OpenLineage/OpenLineage · OpenLineage Blog — Inspiration from OpenTelemetry · USENIX SREcon EMEA 25 — Cross-Platform Data Lineage with OpenLineage · Improving — Effective Data Lineage Strategies for Real-Time Systems · OpenTelemetry Spec Issue #3447 — modeling data lineage in OTel