DEVOURED
Doug Turnbull's experiments show GPT-5-mini with basic BM25 retrieval beats complex search pipelines by 56% on Amazon product data, suggesting most search infrastructure could be replaced by an agent plus a simple backend.
What: Turnbull ran experiments on Amazon ESCI product search where GPT-5-mini agents using only BM25 and E5 embeddings achieved 0.453 NDCG versus 0.289 baseline. The agents typically call search tools once, rewrite queries when results disappoint (e.g. searching 'PVC pipe coupler' after 'pvc coupler' returns network cables), and rank results themselves. Requiring 4+ diverse tool calls pushed performance from 0.410 to 0.431 NDCG. New specialized models like SID-1 train specifically for agentic search workflows.
Why it matters: This works for product/job search where the LLM already understands the domain, but fails for knowledge retrieval. On MSMarco passages, agents showed no improvement over embeddings because the agent can't evaluate information it doesn't know. Two search architectures emerge: thin retrieval plus reasoning agents for catalogs, traditional fitted stacks for research where the LLM has knowledge gaps.
Deep dive
- Baseline comparison on Amazon ESCI product search: BM25 gets 0.289 NDCG, E5 embeddings get 0.314 NDCG
- GPT-5-mini agent with E5 embeddings: 0.359 NDCG, with BM25: 0.385 NDCG, with both tools: 0.410 NDCG
- GPT-5 (larger model) with both tools: 0.453 NDCG, a 56% improvement over baseline with zero model fitting to the data
- Agent behavior: mostly calls each search tool once, retrieves max 20 results, then ranks them internally
- Keyword search prompts more exploration: agents issue 2+ simultaneous queries when BM25 results disappoint (e.g. 'AN10 oil catch can no filter' + 'AN10 oil catch can without breather filter')
- Encouraging exploration: requiring 4 tool calls and blocking duplicate queries improved GPT-5-mini from 0.410 to 0.431 NDCG
- Specialized agentic search models like SID-1 are trained to reason about relevance and explore retrieval thoughtfully, unlike frontier LLMs that assume web search quality and issue single queries
- Domain-specific agentic search models may emerge (e-commerce, job search, document retrieval), similar to embedding model specialization
- MSMarco passages dataset showed no agent improvement over embeddings—agents can't evaluate information they don't possess
- Two search paradigms: 'finding things' (agents excel) vs 'deep research' (traditional stacks needed to compensate for LLM knowledge gaps)
Decoder
- BM25: Classic keyword search algorithm that ranks documents by term frequency and inverse document frequency, standard baseline for text retrieval.
- NDCG (Normalized Discounted Cumulative Gain): Metric measuring search result quality on a 0-1 scale, where higher scores mean more relevant results appear at the top. 0.453 vs 0.289 is a 56% improvement.
- ESCI (Amazon Shopping Queries Dataset): Amazon's public product search dataset pairing customer queries with product relevance labels.
- E5 embeddings: Microsoft's general-purpose text embedding model for semantic search.
- SID-1: Specialized language model trained for agentic search workflows, acts as drop-in replacement for RAG stacks.
- MSMarco: Microsoft's question-answering dataset for passage retrieval, widely used in embedding model training.
Original article
A lightweight LLM agent, given basic retrieval tools (BM25 and/or embeddings), can outperform complex search backends and reranking pipelines, simplifying the search architecture. In experiments on Amazon ESCI data, agentic setups delivered big gains (NDCG from ~0.29 baseline to 0.41-0.45), with agents intelligently rewriting queries, exploring, and evaluating results.
Explore all 73 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi