Puffin File Format

Summary

What it is

A binary format defined inside the Apache Iceberg specification for storing table-level statistics, indexes, and (in V3) deletion vectors as auxiliary blobs alongside Parquet data files. A Puffin file is a sequence of typed blobs plus a footer cataloging blob offsets, sizes, and types.

Where it fits

Puffin is the load-bearing format that turns Iceberg V3 from "Iceberg V2 plus features" into "Iceberg V2 with order-of-magnitude faster MERGE/UPDATE." V3 stores Roaring-bitmap deletion vectors in Puffin blobs instead of rewriting full Parquet data files for every modification.

Misconceptions / Traps

Puffin is not a replacement for Parquet — it is auxiliary. Data files remain Parquet (or ORC); Puffin sits beside them holding indexes and deletion bitmaps.
The Puffin spec is permissive about blob types — engines that don't recognize a blob type just skip it. New blob types can roll out without spec versioning friction.
Backward compatibility is per-blob-type, not per-Puffin-file. An engine reading a Puffin file with both bloom-filter-v1 and deletion-vector-v1 may understand only one.

Key Connections

used_by Iceberg V3 Spec — deletion vectors specifically
used_by Apache Iceberg — table-level NDV and bloom blobs
solves Read / Write Amplification — bitmap deletes replace file rewrites
solves Metadata Overhead at Scale — index/sketch storage separated from data files

Definition

What it is

A binary format defined inside the Apache Iceberg specification for storing **table-level statistics, indexes, and deletion vectors** as auxiliary blobs alongside Parquet data files. A Puffin file is a sequence of typed "blobs" (each tagged with a blob-type identifier such as `apache-datasketches-theta-v1`, `deletion-vector-v1`, `bloom-filter-v1`) plus a small footer that catalogs blob offsets, sizes, and types. Iceberg V3 elevates Puffin from a curiosity to a load-bearing format by storing **Roaring-bitmap deletion vectors** in Puffin blobs instead of rewriting full Parquet data files for every UPDATE/DELETE.

Why it exists

Iceberg V2 supported only positional and equality delete files — both of which required rewriting one or more Parquet files per modification (copy-on-write or merge-on-read with full file rewrites). At CDC scale, that turns every small change into a multi-gigabyte rewrite. Puffin encodes deletes as compact bitmaps over row positions, so a million-row UPDATE writes a kilobyte-scale Puffin blob instead of regenerating data files — yielding **up to 10× faster MERGE/UPDATE** in Iceberg V3 implementations.

Primary use cases

Iceberg V3 deletion vectors for high-frequency CDC and GDPR-style point deletes, table-level NDV (number-of-distinct-values) sketches for query planner optimizations, bloom filters and other index sidecars that accelerate predicate pushdown without bloating the data files themselves.

Recent developments

Latest signals

Puffin Spec. This is a specification for Puffin, a file format designed to store information such as indexes and statistics about data managed in an Iceberg table that ...Read more. Per iceberg.apache.org.
A 2026 Introduction to Apache Iceberg. Deletion Vectors: Binary bitmaps stored in Puffin files that replace position deletes. More compact in storage and faster to apply during reads.Read more. Per dev.to (alexmercedcoder) (2026-02-13).
Question: can I use puffin file to store any customized blobs? · Issue #13011 · apache/iceberg. Puffin file is introduced in iceberg V3, which serves as a container file to store any types of blobs.Read more. Per GitHub (apache/iceberg) (2025-05-08).
Research prototype attaches full ANN vector indexes to Puffin blobs at billion-vector scale. A June 2026 paper extends FlockDB (a distributed MPP engine on DuckDB) to embed Vamana/DiskANN approximate-nearest-neighbor graphs as Puffin sidecar blobs on Iceberg snapshots, requiring only a few hundred lines of table-format integration on top of ~8,000 lines of new ANN code — Puffin's general blob container turns out sufficient to carry graph indexes, not just sketches and deletion vectors. Per Puffin-Backed Vector Indexes (arXiv).
Iceberg V3 (and its Puffin-based deletion vectors) reached GA on Snowflake (May 7, 2026) and Databricks Runtime 18.0+, alongside Iceberg library 1.11.0 (May 19, 2026). Tertiary source, single mention, but consistent with the 1.11 version bump other nodes in this graph are independently tracking. Per Atlan — Apache Iceberg v3 Upgrade Guide.
Iceberg 1.11.0 (May 19, 2026) hardens the deletion-vector write path. The release includes Puffin-specific fixes — notably preventing empty Puffin file creation in the deletion-vector writer — exactly the maturation work you expect once DVs carry production DML traffic. The same release ships the File Format API announced in February 2026, which formalizes how engines plug alternative file formats (and their Puffin-style sidecars) into Iceberg's Java readers and writers. Per Apache Iceberg releases and Introducing the Apache Iceberg File Format API.
Puffin is going polyglot, but engine coverage is still uneven. The official Go implementation now reads and writes Puffin files — the github.com/apache/iceberg-go/puffin package handles DataSketches blobs and deletion vectors outside the JVM, per pkg.go.dev. Meanwhile iceberg-rust still tracks Puffin support as an open epic with reader/writer PRs in flight (apache/iceberg-rust #744), and ClickHouse can read Iceberg V2 delete files but not yet V3 Puffin-based deletion vectors (ClickHouse #107502) — finalized and adopted by major managed platforms, but open-source engine support lags the spec. Sources: iceberg.apache.org · dev.to (alexmercedcoder) · GitHub (apache/iceberg)