Back to Devoured - November 12, 2026
DEVOURED
Parquet's Revised Modular Footer Proposal: Row Groups Stop Being Physical Boundaries

Parquet's Revised Modular Footer Proposal: Row Groups Stop Being Physical Boundaries

Data Apache Parquet Dev Mailing List
Apache Parquet is proposing a modular footer design that separates metadata into independently readable pages, enabling better performance for wide, complex datasets.
What: Led by Jiayi Wang, the community is reviewing a proposal to replace monolithic Thrift-encoded footers with modular, page-based metadata. This change removes the requirement for row groups to be physical boundaries, allowing for granular statistics and more flexible data layouts.
Why it matters: As data warehouses handle increasingly massive and wide tables, the overhead of reading large, static metadata footers becomes a bottleneck; modularity allows for selective metadata retrieval.
Deep dive
  • Shifts footer metadata from a monolithic Thrift block to modular, page-based structures.
  • Eliminates the strict requirement for row groups to be physical data boundaries.
  • Supports optional row-group metadata and statistics at arbitrary row ranges.
  • Enables per-page compression for metadata, reducing read latency for cold data.
  • Improves support for evolving data layouts without requiring constant footer format changes.
Decoder
  • Thrift: A framework and binary protocol developed by Facebook used by Parquet for storing metadata in a compact, cross-language format.
  • Row Group: A subset of data in a Parquet file consisting of column chunks, typically used for pruning scans during queries.
  • Page: The smallest unit of data in a Parquet file, containing values from a single column chunk.
Original article

Hi Jiayi,

I took a pass reviewing. Overall, I like the design. I left a bunch of comments on the footer proto, which I think is pretty dense and could use a cleanup/augmenting of documentation so readers can better piece together how everything fits together.

Thanks, Micah

On Tue, Nov 3, 2026 at 4:21 PM Julien Le Dem via dev wrote:

Thank you Jiayi for leading this effort and everyone who contributed. I would encourage folks on the list to review as it is fairly transformational for the project. I expect that after that we'll avoid changing the footer format again for a while.

On Fri, Oct 30, 2026 at 7:20 AM Jiayi Wang wrote:

Hi everyone,

As mentioned previously, we have revisited the proposal to make it more flexible and also address existing feedback. I’d like to share the new version of the proposal, which is included as a new tab in the same document. A summary of what changed and why:

Flexible Data Layouts

Per discussions on evolving Parquet data layout (e.g. the “non-contiguous pages” proposal), our goal is to have the new footer handle both existing and potential future layouts seamlessly. The updated proposal doesn’t assume that row groups are physical boundaries. It preserves the wording of “column chunks”, but these are simply defined to mean a contiguous set of pages for one column. Different columns may have different numbers of chunks in a file. The “row group” wording is preserved as well, but row group metadata is made optional. Lastly, stats are re-defined based on row ranges rather than row groups; different columns may publish stats at different granularities.

Page-based Representation

A big feedback point on the initial proposal was that the EncodedArray concept was opaque/confusing. It was added to efficiently model arrays in Thrift and provide random access, at the expense of readability. The updated proposal uses Parquet pages to represent modules, which yields a self-describing layout that evolves naturally as Parquet gains new encodings. It also allows for optional per-page compression.

Overall, the new proposal yields a footer that is comparable in size and performance to the original proposal, but is simpler and more flexible for handling future Parquet layout changes. We appreciate any feedback on the proposal.

Best, Jiayi

Proposal 1: https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw

Proposal 2: https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz

Explore all 61 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi