Back to Devoured - May 28, 2026
DEVOURED
Announcing Polars 1.41

Announcing Polars 1.41

Data pythondataframe Polars
Polars 1.41 boosts performance for wide Parquet tables by up to 3.29x and deepens query optimization with nested common subplan elimination.
What: Polars 1.41, released by Thijs Nieuwdorp on May 26, 2026, significantly speeds up Parquet footer decoding, showing a 3.29x improvement for 10,000 columns (from 117.4 ms to 35.74 ms). It also enhances the query optimizer for `LazyFrame` by deduplicating common subplans across all nesting depths and introduces `LazyFrame.gather()` for integer-based row selection.
Why it matters: This update signals Polars' ongoing commitment to high performance and sophisticated query optimization, positioning it as a strong contender against other data manipulation libraries, especially for complex analytical tasks and large datasets.
Takeaway: If you use Polars, upgrade to 1.41 to automatically benefit from faster Parquet scans and more efficient lazy query execution, particularly for wide tables or complex nested operations.
Deep dive
  • Polars 1.41, released on May 26, 2026, focuses on performance and query optimization for data analysis.
  • Parquet metadata decoding is significantly faster, achieving up to 3.29x speedup for tables with 10,000 columns (from 117.4 ms to 35.74 ms).
  • This speedup is due to replacing a generic, auto-generated Thrift decoder with a hand-written, specialized decoder for Parquet's stable metadata schema.
  • The query optimizer now performs common subplan elimination at every nesting depth, preventing redundant computations in complex LazyFrame queries.
  • Previously, subplans shared at higher levels in the query plan might still be re-evaluated; this is now fixed by traversing into cache nodes.
  • LazyFrame.gather() has been introduced, allowing row selection by integer index without first materializing the LazyFrame into a DataFrame.
  • These improvements are automatic for existing scan_parquet calls and LazyFrame operations.
Decoder
  • Polars: A high-performance DataFrame library for Rust and Python, designed for speed and efficiency on large datasets.
  • Parquet: A columnar storage file format optimized for analytics, often used with big data processing frameworks.
  • Thrift: A lightweight, language-independent software stack for point-to-point RPC, used by Parquet to describe its metadata.
  • LazyFrame: A Polars data structure that represents a sequence of operations to be performed later, enabling query optimization.
  • Common subplan elimination: A compiler optimization technique where repeated computations of identical sub-expressions are identified and replaced with a single computation, whose result is then reused.
Original article

Full article content is not available for inline reading.

Read the original article →

Explore all 61 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi