Parquet's Revised Modular Footer Proposal: Row Groups Stop Being Physical Boundaries
Apache Parquet is proposing a modular footer design that separates metadata into independently readable pages, enabling better performance for wide, complex datasets.
Deep dive
- Shifts footer metadata from a monolithic Thrift block to modular, page-based structures.
- Eliminates the strict requirement for row groups to be physical data boundaries.
- Supports optional row-group metadata and statistics at arbitrary row ranges.
- Enables per-page compression for metadata, reducing read latency for cold data.
- Improves support for evolving data layouts without requiring constant footer format changes.
Decoder
- Thrift: A framework and binary protocol developed by Facebook used by Parquet for storing metadata in a compact, cross-language format.
- Row Group: A subset of data in a Parquet file consisting of column chunks, typically used for pruning scans during queries.
- Page: The smallest unit of data in a Parquet file, containing values from a single column chunk.
Original article
Hi Jiayi,
I took a pass reviewing. Overall, I like the design. I left a bunch of comments on the footer proto, which I think is pretty dense and could use a cleanup/augmenting of documentation so readers can better piece together how everything fits together.
Thanks, Micah
On Tue, Nov 3, 2026 at 4:21 PM Julien Le Dem via dev wrote:
Thank you Jiayi for leading this effort and everyone who contributed. I would encourage folks on the list to review as it is fairly transformational for the project. I expect that after that we'll avoid changing the footer format again for a while.
On Fri, Oct 30, 2026 at 7:20 AM Jiayi Wang wrote:
Hi everyone,
As mentioned previously, we have revisited the proposal to make it more flexible and also address existing feedback. I’d like to share the new version of the proposal, which is included as a new tab in the same document. A summary of what changed and why:
Flexible Data Layouts
Per discussions on evolving Parquet data layout (e.g. the “non-contiguous pages” proposal), our goal is to have the new footer handle both existing and potential future layouts seamlessly. The updated proposal doesn’t assume that row groups are physical boundaries. It preserves the wording of “column chunks”, but these are simply defined to mean a contiguous set of pages for one column. Different columns may have different numbers of chunks in a file. The “row group” wording is preserved as well, but row group metadata is made optional. Lastly, stats are re-defined based on row ranges rather than row groups; different columns may publish stats at different granularities.
Page-based Representation
A big feedback point on the initial proposal was that the EncodedArray concept was opaque/confusing. It was added to efficiently model arrays in Thrift and provide random access, at the expense of readability. The updated proposal uses Parquet pages to represent modules, which yields a self-describing layout that evolves naturally as Parquet gains new encodings. It also allows for optional per-page compression.
Overall, the new proposal yields a footer that is comparable in size and performance to the original proposal, but is simpler and more flexible for handling future Parquet layout changes. We appreciate any feedback on the proposal.
Best, Jiayi
Proposal 1: https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw
Proposal 2: https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi