Back to Devoured - November 12, 2026
DEVOURED
Iceberg v4 Stats-Only Fields: How Tables Store Statistics for Partition Expressions

Iceberg v4 Stats-Only Fields: How Tables Store Statistics for Partition Expressions

Data GitHub
Apache Iceberg v4 introduces 'stats-only fields' to track metadata for non-materialized expressions, enabling more precise file pruning without changing table schemas.
What: Merged by Ryan Blue, the v4 specification adds stats-only fields to table metadata. This allows Iceberg to store bounds and statistics for derived values—like bucketed partitions or case-normalized strings—without adding physical columns to the table.
Why it matters: This pattern enables sophisticated query planning and pruning optimizations while keeping the underlying table schema lean and clean.
Deep dive
  • Allows storage of metadata (min/max bounds) for non-materialized, derived expressions.
  • Uses stats-only fields to support partition buckets and computed values (e.g., lowercase transforms).
  • Adds a 'tight_stats' flag to indicate exact row counts and bounds for improved file pruning.
  • Enforces strict rules for writers: they must produce stats only if the expression binding succeeds and yields a compatible output type.
  • Requires readers to gracefully ignore unknown or unsupported stats-only field types.
Decoder
  • Materialization: The process of physically storing the result of an expression as a new table column.
  • Pruning: A query optimization technique where data files that definitely do not match the query predicates are skipped during a scan.
Original article

Full article content is not available for inline reading.

Read the original article →

Explore all 61 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi