Apache DataFusion 54.0.0 Released

We are proud to announce the release of DataFusion 54.0.0. This post highlights some of the major improvements since DataFusion 53.0.0. Notable additions include LATERAL joins, SQL lambda functions, and a new Avro reader, alongside significant join, scan, and planning performance improvements. The complete list of changes is available in the changelog. This release represents roughly 11 weeks of development and 740 commits. Thanks to the 139 contributors (a new record!) for making it possible.

Performance Improvements

We continue to make significant performance improvements in DataFusion, as explained below. This release prunes more redundant work out of plans and makes joins, repartitioning, scans, and many built-in functions faster.

Execution Operator Improvements

Physical Execution of Uncorrelated Scalar Subqueries: DataFusion previously executed an uncorrelated scalar subquery (one that doesn't depend on the outer query) by rewriting it into a join. DataFusion 54 instead evaluates it once with a new physical operator. This lets functions use their specialized scalar code paths, and allows uncorrelated scalar subqueries in ORDER BY, JOIN ON, and as arguments to aggregate functions.

Faster Sort-Merge Joins: Semi, anti, and mark joins now track matches with a per-row bitset instead of materializing (outer, inner) pairs. Batched deferred filtering makes near-unique LEFT and FULL joins 20-50x faster. Finally, join-key comparisons now use a DynComparator that resolves the column type once rather than per row, making microbenchmarks up to 12% faster and TPC-H ~5% faster overall.

Faster Repartitioning: RepartitionExec now coalesces batches before sending them to distributor channels, cutting per-batch overhead for up to 50% faster execution on some repartition-heavy queries.

Faster Functions and Hashing: DataFusion ships hundreds of built-in functions, so speeding them up pays off across many workloads. This release optimizes many, including array_to_string, array_concat, array_sort, split_part, substr, strpos, left, right, string_agg, and approx_distinct, plus better NULL handling across many array and datetime functions. The first_value and last_value aggregates are also substantially faster over Utf8 and Binary columns thanks to a new GroupsAccumulator. DataFusion 54 also swaps ahash for foldhash in datafusion-common, and optimizes regexp_replace by stripping trailing .* from anchored patterns.

Planner Improvements

Pruning Functionally Redundant Sort Keys: Sorting is expensive, so it pays to sort by as few columns as possible. DataFusion 54 now drops functionally redundant ORDER BY keys: when an earlier key determines a later one, the later key can't change the ordering, so removing it cuts sorting cost without affecting results.

Skip Redundant Parquet Filters: When statistics prove a filter matches every row in a Parquet row group, DataFusion now skips evaluating it — both row filters and page-level pruning — for that row group instead of re-checking each row.

Statistics-Driven Sort Pushdown and TopK: Files and Parquet row groups are now ordered using statistics, which can avoid sorting entirely and improve dynamic filtering and early stopping for TopK (ORDER BY ... LIMIT) queries. The most promising data is read first, often satisfying the LIMIT before scanning the rest.

Improved Statistics and Cardinality Estimation: Good plans depend on good statistics. This release extracts NDV (number of distinct values) statistics from Parquet metadata, uses NDV for equality-filter selectivity, adds a pluggable StatisticsRegistry for operator-level statistics propagation, and improves cardinality estimation for semi and anti joins.

Scan Improvements

Morsel-Driven Parquet Scans: Parquet scan parallelism was previously bounded by the slowest scan thread, so data skew (large row groups, less-selective filters, or variable object store latency) left cores underutilized. DataFusion 54 reworks the Parquet scan around a morsel-driven design, where idle threads dynamically pull small units of work ("morsels") instead of each being assigned a fixed partition up front. This spreads work more evenly and can be up to ~2x faster for skewed scans such as ClickBench.

Struct Field Filter Pushdown and Leaf-Level Projection: Filters on struct fields (e.g. WHERE s['foo'] > 67) are now pushed down into the Parquet decoder rather than evaluated after a full scan, and both filtering and projection read only the struct leaves they actually access, significantly improving performance for nested and Variant data in large Parquet files.

New Features

LATERAL Joins

Lateral joins have been long requested. DataFusion 54 adds basic support for CROSS JOIN LATERAL, INNER JOIN LATERAL, and LEFT JOIN LATERAL. A lateral subquery in the FROM clause can reference columns from preceding tables — handy for expanding a per-row series or correlating against a set-returning function. It uses decorrelation, so the subquery is evaluated once rather than re-executed per outer row.

-- For each row in t1, expand a series 1..t1_int and join the values back
SELECT t1_id, t1_name, i
FROM join_t1 t1
CROSS JOIN LATERAL (
    SELECT * FROM unnest(generate_series(1, t1_int))
) AS series(i);

Lambda Functions

DataFusion now supports lambda expressions (x -> expr) with column capture, plus new higher-order array UDFs like array_transform, array_filter, and array_any_match. Lambdas express per-element computation directly in SQL:

-- Apply `x * 10` to every element
SELECT array_transform([1, 2, 3, 4, 5], x -> x * 10);
-- [10, 20, 30, 40, 50]

-- Keep only elements where `x > 2`
SELECT array_filter([1, 2, 3, 4, 5], x -> x > 2);
-- [3, 4, 5]

-- True if any element satisfies `x > 2`
SELECT array_any_match([1, 2, 3], x -> x > 2);
-- true

Spilling Nested Loop Joins

NestedLoopJoinExec previously failed with an out-of-memory error when the build side exceeded the memory budget. DataFusion 54 adds a memory-limited path that transparently spills to disk and completes the query instead, with zero overhead when memory is sufficient. It currently covers INNER, LEFT, LEFT SEMI, LEFT ANTI, and LEFT MARK joins.

New Avro Reader

The Avro reader now uses the arrow-avro crate, replacing internal conversion code with a faster, better-maintained implementation shared with the Arrow ecosystem.

Extension Type Registry

Arrow extension types let users layer their own semantics on top of a physical storage type. DataFusion 54 adds a registry for registering their behavior, several more canonical extension types, and the ability to cast to an extension type in logical expressions.

Content-Defined Chunking for Parquet

DataFusion's Parquet writer can now use content-defined chunking (CDC), which aligns data page boundaries with the data rather than fixed row counts. This improves deduplication and incremental storage, since inserting or editing a few rows no longer shifts every later page boundary.

New Functions

SQL and Scalar Functions: DataFusion 54 adds new scalar functions including array_compact, cosine_distance, inner_product, array_normalize, cast_to_type, and with_metadata, plus nanosecond date_part support and the : JSON access operator. The cosine_distance, inner_product, and array_normalize additions round out DataFusion's vector-search building blocks.

Spark-Compatible Functions: The datafusion-spark crate gains many new or improved Spark-compatible functions, including round, floor, ceil, soundex, xxhash64, array_contains, array_compact, int/float-to-timestamp casts, and UTF-8 validation functions.

Upgrade Guide and Changelog

Upgrading to 54.0.0 should be straightforward for most users, though there are some breaking changes. See the Upgrade Guide for details and migration snippets, and the changelog for the full list of changes.

About DataFusion

Apache DataFusion is an extensible query engine, written in Rust, that uses Apache Arrow as its in-memory format. DataFusion is used by developers to create new, fast, data-centric systems such as databases, dataframe libraries, and machine learning and streaming applications. While DataFusion's primary design goal is to accelerate the creation of other data-centric systems, it provides a reasonable experience directly out of the box as a dataframe library, Python library, and command-line SQL tool.

How to Get Involved

DataFusion is not a project built or driven by a single person, company, or foundation. Rather, our community of users and contributors works together to build a shared technology that none of us could have built alone.

If you are interested in joining us, we would love to have you. You can try out DataFusion on some of your own data and projects and let us know how it goes, contribute suggestions, documentation, bug reports, or a PR with documentation, tests, or code.