DEVOURED
Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion
Apache DataFusion now automatically eliminates redundant sorts by using Parquet min/max statistics to reorder files and prune data at runtime.
What: The update adds 'PushdownSort' to classify scans as Exact or Inexact based on file-level statistics. Exact matches allow removing 'SortExec' entirely, while Inexact matches reorder scans and row groups to tighten Top-K dynamic filters, resulting in 2x–49x speedups for LIMIT queries.
Why it matters: This shows how query engines are evolving to treat data layout as a dynamic optimization target rather than just a storage format, reducing the need for explicit index-based sorting.
Deep dive
- Implements Sort Pushdown to avoid blocking SortExec operators.
- Uses Parquet min/max statistics to prove global ordering for file groups.
- Introduces BufferExec to maintain performance parity in parallel plans without blocking sorts.
- Implements runtime dynamic pruning that uses the Top-K filter to skip row groups across multiple files.
- Upgrades source ordering to 'Exact' if file min/max ranges are non-overlapping and ordered.
- Uses 'Inexact' paths to bias scans towards the most promising data for faster filter convergence.
- Leverages Apache Arrow's RowFilter to skip entire row groups during scanning.
Decoder
- Sort Pushdown: An optimizer rule that moves sorting requirements closer to the data scan, potentially eliminating them.
- Virtual Concatenation: A method of reading multiple sorted files as a single contiguous stream by ensuring ranges do not overlap.
- Top-K: A query pattern returning the N largest or smallest values, optimized by tracking the threshold of the Nth value seen so far.
Original article
Full article content is not available for inline reading.
Explore all 56 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi