Write-Audit-Publish
A data quality pattern where data lands in a raw S3 zone, undergoes validation, and is promoted to a curated zone only after passing audits.
Summary
A data quality pattern where data lands in a raw S3 zone, undergoes validation, and is promoted to a curated zone only after passing audits.
WAP is the quality gate for S3 data lakes. It prevents bad data from reaching production consumers by isolating writes in a staging area, running validation checks, and only publishing data that passes.
- WAP requires branching or snapshot isolation. Without a table format that supports branches (Iceberg) or staging areas (lakeFS), implementing WAP on raw S3 is manual and error-prone.
- Audit logic must be idempotent. If audits fail and data is re-submitted, the system must handle duplicates gracefully.
depends_onS3 API — data lands in S3 for stagingsolvesSchema Evolution — catches incompatible changes before they affect consumersscoped_toData Lake, S3
Definition
A data quality pattern where incoming data lands in a raw S3 zone, undergoes automated validation (schema checks, data quality rules), and is promoted to a curated zone only after passing audits.
Publishing unchecked data directly into production zones causes downstream breakage. This pattern gates promotion behind validation, ensuring only quality data reaches consumers.
Data lake quality gates, regulated data pipelines, self-service data onboarding with automated validation.
Recent developments
- Apache Iceberg branching makes WAP native —
wap.branchproperty is the 2026 production default. Iceberg's branch-level WAP property lets engines write to an isolated audit branch, validate, then atomically fast-forward main to the validated commit. Replaces the older "audit staging table" pattern with metadata-only operations. Per Dremio — Streamlining Data Quality in Apache Iceberg with WAP & Branching. - AWS published canonical reference: Iceberg branching + Glue Data Quality. AWS Big Data Blog's reference architecture pairs Iceberg's audit branch with AWS Glue Data Quality validation rules — the audit step runs against the branch, publish only happens after a Glue DQ pass. First major-cloud-vendor canonical pattern. Per AWS Big Data Blog — Build WAP with Iceberg branching and Glue Data Quality.
- Three-step audit pattern: NULL detection + duplicate checks + schema integrity. The 2026 minimum audit set has consolidated around these three checks. Add freshness + range + cardinality for higher-stakes pipelines. Per Telmai Blog — What is Write-Audit-Publish in Apache Iceberg.
- Expedia public engineering write-up: production WAP at scale. Expedia documented their production WAP rollout — branch-per-pipeline-run + DQ validators + automated rollback. Hands-on adoption signal that the pattern is production-mature, not just blog-post architecture. Per Expedia Group Tech — Chill Your Data with Iceberg Write-Audit-Publish.
- Publish step is now a metadata-only atomic commit. WAP's "publish" used to require physically moving data; with Iceberg V3 branching it's a single
fast-forwardsnapshot operation — the data already exists in the branch, only the metadata pointer moves. Drastically reduces publish-latency and failure modes. Per Hashnode — Write → Audit → Publish (WAP) pattern with Iceberg. - Version-controlled + testable + auditable pipelines is the 2026 framing. WAP is reframed in 2026 not just as data-quality but as the "Git for data pipelines" pattern — isolated branches, atomic publishes, version-controlled state. Operational discipline catches up to software-engineering discipline. Per Substack — Write-Audit-Publish (WAP) Pattern by Julien Hurault.
Connections 6
Outbound 4
Inbound 2
enables2Resources 3
lakeFS's definitive engineering blog on the WAP pattern, explaining the write-audit-publish workflow with practical implementation guidance for data lakes.
Apache Iceberg documentation covers branch-based WAP workflows natively, with hidden snapshots that enable the audit step before publishing to production.
Analysis of how Netflix implements WAP using Iceberg's hidden snapshots and internal auditor tools to guarantee data quality.