Technology

Apache Paimon

An Apache top-level streaming lakehouse table format built on LSM-tree architecture, designed for high-frequency real-time writes and sub-minute data visibility on object storage.

10 connections 3 resources

Summary

What it is

An Apache top-level streaming lakehouse table format built on LSM-tree architecture, designed for high-frequency real-time writes and sub-minute data visibility on object storage.

Where it fits

While Iceberg and Delta focus on batch-first with streaming bolted on, Paimon is streaming-first. Its LSM-tree design on S3 enables minute-level data visibility for CDC workloads, making it the natural choice for Flink-based real-time pipelines writing to object storage.

Misconceptions / Traps

Paimon's strength is Flink integration. Spark support is improving but lags significantly behind Flink in maturity and performance.
Higher metadata complexity than Iceberg. The LSM-tree compaction process adds operational overhead that batch-oriented formats do not have.

Key Connections

depends_on S3 API — stores data as objects on S3
depends_on Apache Parquet — data file format
enables Lakehouse Architecture — streaming-first lakehouse design
competes_with Apache Hudi — both target real-time ingestion workloads

Definition

What it is

An Apache top-level table format built on LSM-tree (Log-Structured Merge-tree) architecture, designed for high-frequency streaming writes and real-time analytics on object storage. Originally developed as Flink Table Store.

Why it exists

Traditional lakehouse table formats (Iceberg, Delta, Hudi) were designed primarily for batch workloads with streaming bolted on. Paimon is built streaming-first, using LSM-trees on S3 to enable minute-level data visibility for CDC and real-time analytics without the write amplification penalty of copy-on-write.

Primary use cases

Real-time CDC ingestion into the lakehouse, streaming analytics with minute-level visibility, high-frequency update workloads on S3.