Vector Search on Apache Iceberg Without Moving Source Data

Back to Devoured - November 02, 2026

DEVOURED

Vector Search on Apache Iceberg Without Moving Source Data

Data infrastructureai StarTree

StarTree now supports vector search on Apache Iceberg by serving a secondary index without duplicating source records.

What: StarTree leverages Pinot's HNSW vector index technology to enable vector search directly on Parquet files stored in object storage, keeping indices distinct from the raw data.

Why it matters: This architecture avoids the common overhead of migrating datasets into specialized vector databases while maintaining support for hybrid SQL and similarity search.

Deep dive

Pinot's HNSW index is generated as a sidecar to existing Iceberg Parquet files.
Queries perform hybrid similarity search combined with SQL filters natively.
Data freshness is tied to Iceberg snapshot synchronization, not real-time streaming.
Trade-off includes additional storage usage proportional to embedding size.

Decoder

HNSW (Hierarchical Navigable Small World): A graph-based indexing algorithm commonly used for approximate nearest neighbor search.
Apache Iceberg: An open table format for large-scale analytic datasets that tracks files and snapshots to provide high-performance query execution.

Original article

StarTree adapts Pinot's HNSW vector index for Iceberg tables, keeping Parquet source records in object storage while storing and serving a derived index alongside them. This enables hybrid similarity and SQL-filtered queries without a second copy of the records. The trade-off: indexes are roughly as large as the embeddings, and freshness follows Iceberg snapshot synchronization rather than real-time ingestion.

Explore all 69 articles in today's digest