DEVOURED
Primary Keys Come to Apache Iceberg V4, and Most of Them Start Out Unenforced
Apache Iceberg V4 introduces primary keys and check constraints, but initially treats them as informational metadata rather than enforced write-time rules.
What: The Iceberg community approved the V4 spec, adding PRIMARY KEY, UNIQUE, and CHECK constraints that track validation status per snapshot.
Why it matters: This acknowledges the reality of lakehouse environments where multiple engines (Spark, Flink) write to the same table; it allows teams to declare rules and track compliance without mandating consistent write-time enforcement across all engines.
Takeaway: Start inventorying your tables' natural keys and nullability today, and run validation queries on V3 tables to prepare for eventual V4 upgrades.
Deep dive
- V4 spec adds CHECK, UNIQUE, and PRIMARY KEY constraints.
- Constraints are tracked via IDs in table metadata, separate from the schema.
- Every snapshot records a list of constraint IDs that are 'validated', 'valid', 'invalid', or 'unvalidated'.
- Enforcement is a 'promise' by writers; if enforced: false, writers ignore it, but readers can still use the metadata status.
- V4 forbids equality deletes, pushing upserts toward position-based mechanisms.
Decoder
- Iceberg Snapshot: An immutable state of an Iceberg table at a point in time.
- Equality Delete: An Iceberg mechanism that deletes rows based on a condition rather than a file position; now deprecated for V4.
Original article
Full article content is not available for inline reading.
Explore all 69 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi