DEVOURED
Apache DataFusion 55.0.0 Released
Apache DataFusion 55.0.0 arrives with significant performance gains, native range partitioning, and long-awaited SQL MERGE INTO support.
What: This release includes improvements from 175 contributors, featuring runtime row-group pruning, optimized TopK evaluation, and a pluggable spill backend architecture. It also adds file-level metadata access functions and reduces Parquet read overhead for nested columns.
Why it matters: As DataFusion continues to mature as a query engine foundation for Rust-based data tools, the focus on extensibility—such as custom spill backends and FFI-friendly planning—demonstrates a clear move toward supporting more complex, distributed analytical systems.
Takeaway: If you are using DataFusion for Parquet processing, upgrade to 55.0.0 to benefit from improved predicate pushdown and reduced memory usage.
Deep dive
- MERGE INTO support added with TableProvider hooks for Iceberg/Delta Lake integration.
- Native range partitioning avoids expensive re-shuffling for time-series workloads.
- Runtime row-group pruning for TopK queries speeds up ORDER BY ... LIMIT operations by 4.2x.
- Parquet nested-column pruning reduces I/O by avoiding unnecessary decoding of unused fields.
- Pluggable spill backends allow routing disk spills to custom storage layers like object stores or database buffer pools.
- New file_row_index() and input_file_name() functions aid in debugging and data lineage.
Decoder
- Parquet: A columnar storage file format used in big data systems.
- Spill: Writing data to disk when memory usage exceeds a configured threshold.
- FFI (Foreign Function Interface): A mechanism for a program written in one language to call functions or use services written in another language.
Original article
Full article content is not available for inline reading.
Explore all 65 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi