Back to Devoured - November 16, 2026
DEVOURED
How Open Is Open? Iceberg, Catalogs, and User Choice

How Open Is Open? Iceberg, Catalogs, and User Choice

Data Data Engineering Weekly
True interoperability in data platforms requires a community-maintained baseline for catalogs, not just open table formats like Iceberg.
What: Ananth Packkildurai argues that Apache Iceberg users face hidden vendor lock-in via catalog implementations. He proposes a community baseline using the REST catalog protocol, Apache Polaris, and public compatibility testing to ensure portability.
Why it matters: Vendor-specific catalog features can hide proprietary logic, making it difficult for teams to switch compute engines or storage providers without platform rebuilds.
Takeaway: Advocate for using an open catalog implementation like Apache Polaris to maintain portability across different query engines.
Deep dive
  • Iceberg's table format is open, but catalog behavior is often vendor-specific.
  • Catalog dependencies include commit logic, metadata handling, and credential management.
  • The REST catalog protocol is the current target for standardizing engine-catalog interaction.
  • The community should adopt a "REST Compatibility Kit" for testing.
  • Standardized benchmarking for metadata loading and concurrent writes is needed to compare vendors fairly.
Decoder
  • Catalog: The component in a data lakehouse that maintains table metadata, tracks versions, and manages concurrency.
  • Table format: A specification (like Iceberg, Delta Lake, or Hudi) that manages how data files are organized and how metadata is stored.
Original article

How Open Is Open? Iceberg, Catalogs, and User Choice

The next test of openness: giving users the freedom to choose engines, control their catalogs, and change providers.

Building a vendor-neutral data platform has long been a dream for companies: one free of vendor lock-in, with better cost control, and one that enables faster analytical insights. A simple and fair ask. We all know the reality is different, even with “open” table formats.

Recent debates about Apache Iceberg’s openness and interoperability have reinforced a concern I believe the industry must address:

users need a definition of openness they can verify independently of the vendors making the claim.

For data engineering leaders, openness carries architectural and financial consequences. Can a team change query engines, operate its own catalog, or replace a provider without rebuilding its platform? What capabilities, performance, or control would it lose?

Apache Iceberg provides a key foundation for that independence. Iceberg’s REST catalog protocol puts commit logic behind a common interface and supports mechanisms to delegate storage access. The catalog therefore influences how engines access and update tables. Its operator occupies a consequential position in the architecture.

That position creates commercial influence. Vendors invest in integrations, security, and performance to make their platforms attractive. Users need to understand when those advantages create dependencies that constrain future choices.

A published interface establishes a contract; a shared implementation and reproducible tests help expose its practical boundaries.

Connecting an engine to a catalog proves only part of the interoperability story. Production introduces growing metadata, concurrent writers, credential renewal, and demanding workloads. Iceberg supports features like lazy snapshot loading and server-side scan planning. Engine and catalog implementations can use these capabilities differently, affecting efficiency before data processing begins.

Better engineering should deliver better performance. Users need evidence to distinguish those improvements from incomplete integrations, unsupported features, and proprietary dependencies. Otherwise, every organization must independently investigate the boundaries of openness.

The Iceberg community should establish a clear operational baseline and encourage the ecosystem to adopt it.

Hive Metastore offers a useful precedent. It gave engine developers a concrete integration target and organizations a service they could operate. That common foundation helped developers reproduce behavior and users investigate problems outside a vendor’s environment.

The community already has substantial work to build on. Apache Polaris implements Iceberg’s REST API, and Iceberg maintains a REST Compatibility Kit for testing catalog servers. The next step should connect these contributions to explicit expectations for adoption, validation, and user choice.

The community should rally around a production-ready catalog that users can deploy independently, and engines can support as a common integration target.

It should welcome alternative implementations while keeping the specification authoritative. Public, versioned compatibility results should identify supported operations, optional capabilities, configurations, and limitations.

Performance workloads should complement those results. With controlled data, compute, network placement, and cache conditions, users could compare metadata loading, planning behavior, and concurrent commits. Those comparisons would explain performance differences and help teams make informed decisions.

Migration must receive equal attention. Changing catalogs involves permissions, identities, credentials, catalog-specific state, and active writers. The community should publish tested procedures that explain what users can move and which dependencies remain.

A dependable baseline would strengthen competition. Vendors could demonstrate improvements against shared expectations. Users could evaluate the benefits and dependencies together. Engine developers could integrate against a foundation the community maintains and tests.

The public debate exposed a question that deserves an engineering answer. The Iceberg community should turn that question into shared code, published evidence, and practical guidance.

“How open is open?” should come down to what users can run, what they can change, and what they can take with them. The community must give them the evidence to make that choice.

Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi