DEVOURED
How we built Elasticsearch simdvec to make vector search one of the fastest in the world
Elasticsearch built a custom SIMD kernel library that speeds up vector distance computations by up to 4x through bulk scoring and memory latency hiding techniques.
What: simdvec is a hand-tuned SIMD kernel library written in native C++ and called from Java via the Panama foreign function interface that powers all vector distance computations in Elasticsearch. It supports multiple vector types (float32, int8, bfloat16, binary, and BBQ quantization) and provides architecture-specific optimizations for x86 (AVX-512) and ARM (NEON).
Why it matters: The biggest performance gain comes not from faster arithmetic but from hiding memory latency at production scale—when vector datasets exceed CPU cache limits, simdvec's bulk scoring with explicit prefetching on x86 or interleaved loading on ARM keeps the CPU pipeline busy while waiting for DRAM access, which is where real-world vector search workloads actually operate.
Takeaway: If you're building vector search systems or doing high-volume distance computations, design for bulk scoring APIs and explicit memory management over single-pair operations—the architecture matters more than raw SIMD speed once data exceeds cache.
Deep dive
- Elasticsearch built simdvec because existing solutions (Panama Vector API, FAISS, jvector) couldn't cover the full range of quantized vector types and bulk scoring requirements needed for production search workloads
- The library provides hand-tuned native C++ kernels for every vector type Elasticsearch supports, with single-digit nanosecond FFI call overhead from Java
- On single-pair float32 scoring at 1024 dimensions, simdvec is competitive with FAISS on x86 (28ns vs 23ns) and leads on ARM (70ns vs 156ns for FAISS, 110ns for jvector)
- For int8 quantization, simdvec matches NumKong performance while supporting far more vector types through unified FFI interface
- The real advantage emerges with bulk scoring: when scoring thousands of vectors with working sets exceeding L3 cache, simdvec outperforms alternatives by 1.7x to 4x
- On x86, explicit prefetching reduces cache misses from 139K to 19K and doubles instructions per cycle, with performance advantage growing from 1.2x in L2 to 2.8x beyond L3 cache
- On ARM, interleaved loading from four vectors simultaneously provides memory-level parallelism, reducing backend stalls by 40% with consistent 1.8x speedup regardless of dataset size
- Two architectures require fundamentally different strategies: x86 uses sequential processing with prefetch instructions to pull future data into L1, while ARM uses interleaved access patterns to keep the out-of-order engine fed
- Production vector indices typically don't fit in CPU cache (a 10M-vector int8 index at 1024 dimensions is 10GB), making memory latency hiding the dominant performance factor
- Every vector search query—whether HNSW graph traversal, IVF candidate scoring, or reranking—executes millions of distance operations through the same simdvec engine, compounding kernel-level improvements into significant query latency and throughput gains
Decoder
- SIMD: Single Instruction Multiple Data, a CPU feature that performs the same operation on multiple data points simultaneously using wide registers (256-bit or 512-bit)
- AVX-512: Advanced Vector Extensions 512-bit, Intel's SIMD instruction set for x86 processors that can process 16 float32 values in parallel
- NEON: ARM's SIMD instruction set architecture for parallel processing on ARM processors like Graviton
- HNSW: Hierarchical Navigable Small World, a graph-based approximate nearest neighbor search algorithm
- IVF: Inverted File index, a clustering-based search structure that partitions vectors into buckets
- BBQ: Better Binary Quantization, an aggressive quantization technique that reduces memory by 32x
- FFI: Foreign Function Interface, a mechanism for calling native C/C++ code from higher-level languages like Java
- Panama: Java's Project Panama, which provides FFI capabilities and vector API for SIMD operations
- Prefetching: Issuing CPU instructions to load data into cache before it's actually needed, hiding memory latency
- Cache miss: When requested data isn't in CPU cache and must be fetched from slower DRAM, causing pipeline stalls
Original article
Elasticsearch's simdvec is a hand-tuned SIMD kernel library that accelerates vector distance computations across all query types, using techniques like bulk scoring, prefetching, and architecture-specific optimizations to significantly outperform alternatives—especially at large scale when data exceeds CPU cache. Its biggest advantage comes not from raw compute speed but from efficiently hiding memory latency, enabling faster, more scalable vector search across diverse data types and hardware.
Explore all 66 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi