Back to Devoured - May 07, 2026
DEVOURED
S3 is the perfect place to store data, until you try to search it

S3 is the perfect place to store data, until you try to search it

Data opensourceinfrastructuredatabaserust
Gordon Murray built Firn, an open-source vector search API for S3, after discovering Turbopuffer (which handles 3.5 trillion documents) is closed-source SaaS-only — achieving 72-microsecond cache-hit latency using Lance format and foyer cache from RisingWave.
What: Gordon Murray created Firn (github.com/gordonmurray/firnflow), an open-source API for vector and full-text search on S3-backed data. Built with Axum (Rust), Lance columnar format, and foyer cache, it delivers 72-microsecond latency for repeated queries and ~1 second for fresh queries. He solved cache invalidation with a generation counter and concurrent writes using S3's If-None-Match header. Stress testing across 8 object storage providers revealed that Google Cloud Storage silently ignores conditional writes and Backblaze B2 doesn't support them, while Tigris fixed a data-loss bug over a weekend after Murray reported it.
Why it matters: This exposes a gap in the open-source ecosystem for S3-native vector search and demonstrates that intelligent caching can make object storage viable for low-latency workloads previously requiring expensive managed services like OpenSearch. The conditional-write testing across providers also reveals significant compatibility differences that could silently corrupt data in production.
Takeaway: If you need vector search without expensive managed services, try Firn on GitHub (gordonmurray/firnflow) — it works with AWS S3, MinIO, Cloudflare R2, Tigris, and DigitalOcean Spaces, but avoid Google Cloud Storage and Backblaze B2 for this use case.
Deep dive
  • Gordon Murray created Firn after discovering Turbopuffer (serverless vector/full-text search handling 3.5 trillion documents in production) is closed-source SaaS-only with no free tier
  • Built with Axum (Rust HTTP framework), Lance columnar format, and foyer cache (originally developed inside RisingWave for S3 latency reduction)
  • Performance evolution: 25 seconds per query on raw S3 → 1 second with Lance indexing → 72 microseconds for cache hits (fresh queries still take ~1 second; cache only helps repeated queries keyed by query hash)
  • Solved cache invalidation with generation counter: per-tenant value bumped on every write makes old cache entries unreachable without deletion overhead that scales with data size
  • Solved concurrent writes using S3's If-None-Match:
  • header for conditional PUTs, preventing silent overwrites when multiple writers target the same namespace
  • Fragment problem: Lance appends a new file on each save, so 500 saves = 500 files to open per query; solved with /compact endpoint that merges fragments into optimized files
  • Tested 8 object storage providers for conditional write support: AWS S3, MinIO, Cloudflare R2, Tigris, and DigitalOcean Spaces passed; Google Cloud Storage silently ignores If-None-Match (returns 200 OK on second conditional PUT), Backblaze B2 returns 501 NotImplemented
  • Tigris initially failed 8-writer concurrent stress test with row loss on both dual-region and single-region buckets; Ovais Tariq from Tigris fixed the bug over the weekend and passed 100/100 iterations on Monday
  • Automatic schema inference: vector dimension detected from first write or existing S3 manifest, allowing single Firn instance to host different models side-by-side (e.g., 128-dim image search + 1536-dim text search) without global configuration
  • Real-world deployment: Metabare.com (minimal image search engine) migrated from local EBS storage to Firn backend for cost reduction
  • Exposes Prometheus metrics like s3_requests_total to track cache hits and quantify S3 cost reduction (every cache hit avoids an S3 request)
  • Roadmap includes auto-indexing after N rows and background compaction to eliminate manual triggers
Decoder
  • Lance: Columnar data format optimized for ML/AI workloads, designed for efficient vector storage and random access on object storage
  • foyer: SSD and RAM cache library originally developed inside RisingWave, designed to reduce object storage latency to memory-level speeds
  • BM25: Best Match 25, a ranking function used for full-text search scoring
  • **If-None-Match: ***: HTTP header for conditional writes that only succeeds if the object doesn't already exist, acting as a lightweight distributed lock
  • Generation counter: Versioning technique that invalidates cache by incrementing a namespace value rather than deleting entries, providing constant-time invalidation regardless of data size
  • Fragment problem: In append-only columnar formats like Lance, each write creates a new file fragment, causing queries to open and scan hundreds of small files instead of a few optimized ones
Original article

Firn is an open-source API for fast vector and full-text search on S3-backed data, using Lance plus caching to make repeated queries extremely fast. It's useful for teams that want searchable object storage without the cost or complexity of running OpenSearch.

Explore all 73 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi