Summary

What it is

A distributed stream processing framework that processes data in real-time, with S3 as checkpoint store, state backend, and output sink.

Where it fits

Flink is the streaming complement to Spark's batch processing. In the S3 world, Flink continuously ingests data into lakehouse tables (Iceberg, Delta) and uses S3 for fault-tolerant checkpointing.

Misconceptions / Traps

Flink streaming writes to S3 inherently produce small files (one file per checkpoint interval per writer). Compaction is mandatory — either via the table format or a separate job.
Flink's S3 filesystem plugin requires careful configuration. The wrong S3 filesystem implementation (s3:// vs s3a:// vs s3p://) causes silent failures.

Key Connections

used_by Medallion Architecture, Lakehouse Architecture — streaming data into lakehouse layers
constrained_by Small Files Problem — streaming writes produce many small files
scoped_to S3, Data Lake

Definition

What it is

A distributed stream processing framework that processes data in real-time, with S3 serving as a checkpoint store, state backend, and output sink.

Why it exists

Batch processing alone cannot satisfy requirements for fresh data. Flink enables continuous processing of streaming data, with S3 as the durable layer for checkpoints (fault tolerance) and as the final destination for processed output.

Primary use cases

Real-time data ingestion into S3-backed lakehouses, streaming ETL with S3 sink, checkpoint storage on S3 for fault tolerance.

Recent developments

Latest signals

Flink 1.20.4 ships 41 bug fixes; Flink 2.3 lining up behind it. Per the Apache Flink 1.20.4 release announcement (April 22, 2026), the 1.20 LTS line continues to receive maintenance with 41 bug fixes and minor improvements. The April 2026 community update flags Flink 2.3 as the next major release, with Materialized Tables work, Flink CDC 3.6.0, and a Flink Agents 0.2.1 line that brings agentic-LLM execution into the Flink runtime — the project is broadening from "stream processor" toward "stream-native compute substrate" for AI workloads.
Native S3 FileSystem benchmarks at ~2× Presto S3 for checkpoint throughput. Per the Apache Flink wiki benchmark (February 2026), Native S3 checkpoints sustain ~190 MB/s versus Presto S3 at ~89 MB/s — a 2× advantage that matters most for stateful jobs with high checkpoint frequency. Combined with ForSt (Flink 2.0's tiered state backend), recovery times drop below 10 seconds even for large stateful jobs per RisingWave's Flink comparison — the structural bet against having to keep all state in RAM is paying off.
Alibaba's Realtime Compute for Flink lands AIOps + fine-grained permissions (April 20, 2026). Per Alibaba Cloud's release notes, the managed Realtime Compute service now ships AIOps integration, end-to-end observability, and fine-grained permission controls — productizing Flink for the enterprise-governance shape that Databricks and Snowflake have set for the analytical side. The competitive frame: managed Flink is being repositioned from "compute primitive" to "governed platform tier."
CVE-2026-35194 — RCE via SQL injection in Flink's code generation, quietly patched by "bug fix" releases already cited in this profile. User-controlled strings get interpolated into generated Java code without escaping; affects 1.15.0 through 1.20.x and 2.0.0 through 2.x. Fixed in 1.20.4, 2.0.2, 2.1.2, and 2.2.1 — the same versions this profile already lists as routine maintenance were in fact security-critical upgrades. Tertiary source. Per SentinelOne — CVE-2026-35194.
Release cadence continued past this profile's April snapshot. Flink 1.20.5 (June 3), 2.0.2 (May 11), 2.2.1 (May 15), and 2.1.3 (June 14) all shipped; Flink Kafka Connector 5.0.0 landed June 2, and Flink Kubernetes Operator 1.15.0 followed on May 26. Per Apache Flink — Downloads and Apache Flink project news.
Stateful Functions is being sunsetted. The community is retiring the StateFun sub-project — teams running event-driven stateful services on it should plan a migration path. The project's application-layer energy has shifted to Flink Agents, the sub-project bringing agentic-LLM execution into the runtime. Per Flink Community Update for April 2026 (flink.apache.org).
Ververica (Flink's original commercial steward) is sunsetting its Cloud Managed Service to prioritize sovereign BYOC deployments. Platform 3.1 shipped April 2026; the new pitch is "Streamhouse" — continuous data flow replacing batch. Tertiary source (company blog; unattributed 2x-perf/40%-lower-TCO claims — weight accordingly). Per Ververica Blog. Sources: Apache Flink 1.20.4 release (April 22, 2026) · Flink Community Update — April 2026 · Native S3 FileSystem vs Presto S3 benchmark (Apache wiki) · Realtime Compute for Apache Flink — April 20, 2026 release (Alibaba Cloud) · RisingWave vs Apache Flink comparison