Apache Flink 2.3.0 Release Announcement

The Apache Flink PMC is pleased to announce the release of Apache Flink 2.3.0.

This release significantly expands SQL capabilities with changelog conversion operators, enhances materialized table flexibility, introduces an experimental, high-performance native S3 filesystem, and delivers application management. Flink 2.3.0 brings together contributors from around the globe, implements full or core functionalities of 15 FLIPs (Flink Improvement Proposals), and resolves numerous issues and enhancements.

Key improvements in this release include new SQL operators for changelog manipulation (FROM_CHANGELOG and TO_CHANGELOG), fine-grained control over materialized table refresh strategies, adaptive partition selection for optimized backpressure handling, and a completely redesigned S3 filesystem built on AWS SDK v2. The introduction of application-level lifecycle management provides better visibility and control for production deployments, while enhanced watermark alignment can dramatically improve backlog processing performance. A reworked SinkUpsertMaterializer brings much improved performance for some Flink SQL workloads.

We extend our heartfelt thanks to all contributors for making this release possible!

Flink SQL Improvements

FROM_CHANGELOG and TO_CHANGELOG: Bridging Append-only and Dynamic Changelog Tables

The DataStream API has long offered toChangelogStream() and fromChangelogStream() for working with changelog streams; Flink 2.3 brings equivalent functionality to SQL via two new built-in Process Table Functions:

• FROM_CHANGELOG converts an append-only stream that carries an operation column into a dynamic table. A configurable op_mapping makes it straightforward to plug in custom CDC formats and controls how rows with unmapped operation codes are treated.
• TO_CHANGELOG is the inverse: it materializes a dynamic table back into an append-only changelog stream. This is the first SQL-level operator that lets users convert retract or upsert streams into append form — useful for archival, audit, writing to append-only sinks, and working around pipelines that require an append-only table.

The 2.3 release covers limited basic use cases for both. Future versions will extend both functions with PARTITION BY, invalid_op_handling, produces_full_deletes and more to make both features powerful and extensive.

Materialized Table Evolution: DDL Parity and Refresh Control

Flink 2.3 brings materialized tables to feature parity with regular tables through two major enhancements.

First, CREATE MATERIALIZED TABLE now accepts explicit column definitions, including watermarks and primary keys, just like regular tables. ALTER MATERIALIZED TABLE gains full DDL capabilities—ADD, MODIFY, and DROP operations for metadata and computed columns, plus RENAME TO, allowing materialized tables to evolve through the same workflow already used for regular Flink tables.

Second, Flink 2.3 introduces granular control over data reprocessing when a materialized table’s query changes. The new START_MODE clause lets you choose exactly where the refresh pipeline begins. There is also special support for attempting to resume processing from the exact source offsets where the previous job instance stopped.

These enhancements eliminate the need to drop and recreate materialized tables when query definitions evolve, and prevent unnecessary reprocessing of historical data when iterating on pipeline logic.

SinkUpsertMaterializer: Explicit Conflict Handling

The SinkUpsertMaterializer is required when the upsert key (the unique identifier provided by the stream) is different from the primary key (the unique identifier in the target sink table). This happens in scenarios like multi-stage transformations, projections, or joins.

By default, queries now fail at planning time when upsert and primary keys differ, requiring you to explicitly choose a conflict strategy. This is done with a new ON CONFLICT clause that makes the behavior explicit. You choose how to handle conflicts: DO NOTHING (silent skip), DO ERROR (fail the job), or DO DEDUPLICATE (materialize and deduplicate, similar to what Flink has done until now):

INSERT INTO target_table
SELECT * FROM source
ON CONFLICT DO DEDUPLICATE;

Second, watermark-based compaction reduces state size by cleaning up old changelog records that can no longer affect the final result. Two new configuration options control the compaction behavior:

• table.exec.sink.upserts.compaction-mode (default: WATERMARK) — WATERMARK or CHECKPOINT.
• table.exec.sink.upserts.compaction-interval — optional fallback interval for emitting watermarks when none arrive naturally.

Process Table Function Enhancements

Process Table Functions (PTFs), introduced in Flink 2.1, gain new capabilities that align them with the DataStream API:

• Late data handling: PTFs can now react to late records instead of silently dropping them, enabling custom late data strategies at the SQL level.
• Ordered table arguments: The new ORDER BY clause on table arguments ensures PTFs receive rows in deterministic temporal order within each partition:

SELECT * FROM 
  MyTimestampedPtf(
    input => TABLE events PARTITION BY user_id ORDER BY event_time
  );

ARTIFACT Keyword for User-Defined Functions

The USING clause of CREATE FUNCTION now accepts an ARTIFACT keyword as a future-proof alternative to JAR. This generic keyword prepares the syntax for future ecosystem assets like Python wheels, while remaining fully backward compatible.

Critical Bug Fix: MiniBatch Aggregation Record Loss

Flink 2.3 fixes a critical bug in MiniBatchGroupAggFunction that could silently drop records when mini-batch aggregation was enabled and the planner used a ONE_PHASE aggregation strategy.

Connectors

Native S3 FileSystem

Flink 2.3 introduces a ground-up S3 Filesystem with flink-s3-fs-native, a new plugin built directly on AWS SDK v2. This Native S3 FileSystem is experimental in Flink 2.3. It is functionally complete, and replaces the Hadoop and Presto-based connectors with a modern implementation that delivers:

• Better performance
• Native AWS integration: IAM Roles for Service Accounts (IRSA), modern credential providers, and direct SDK v2 integration
• Non-blocking I/O: Asynchronous operations for improved throughput
• Unified implementation: Single plugin provides both FileSystem and RecoverableWriter
• Zero Hadoop dependencies: No dependency mess with smaller footprint

Runtime Improvements

Support Adaptive Partition Selection

For the RebalancePartitioner and RescalePartitioner data partitioning modes, Flink 2.3 introduces an adaptive data partitioning feature based on downstream task load. It distributes data dynamically according to downstream workloads to balance traffic and improve job throughput.

AdaptiveScheduler Rescale History and Web UI

Streaming jobs using the adaptive scheduler now record a complete history of rescale events, including parallelism changes, slot allocations, scheduler state transitions, and termination reasons. This data is available through new REST endpoints and visualized in a dedicated “Rescales” tab in the Flink Web UI.

Watermark Alignment for Fast Backlog Processing

In Flink 2.3 the watermark alignment was redesigned to solve announcement delays by the introduction of a watermark alignment buffer. By default this buffer has a size of 3 and it delays the application of the watermark alignment algorithm by 3 update intervals.

Checkpointing During Recovery

Flink now supports triggering checkpoints while a job is still recovering from unaligned checkpoints. Previously, a checkpoint could only be triggered after all restored channel state had been fully consumed.

Application-Level Lifecycle Management

Flink 2.3 introduces a first-class application concept that unifies behavior across deployment modes. The cluster-job model is replaced by a cluster-application-job hierarchy, providing better visibility and control for production deployments.

Robust OpenTelemetry Metrics Export

Jobs with many tasks and operators can produce metric payloads large enough to be rejected by OTel gRPC backends, causing metric loss in production. Flink 2.3 adds two robustness features to the OTel exporter: gzip compression and batching.

Documentation

The Flink documentation has been reorganized to make navigation easier, with Flink SQL getting a dedicated top-level section, relational streaming concepts promoted to a top-level Concepts section, and Python documentation integrated into API sections.