DEVOURED
Q2D-Web: Evaluating First-Stage Retrievers at Scale
Q2D-Web introduces a large-scale benchmark containing 190 million documents to evaluate how web search retrieval models perform across ten languages.
What: Perplexity developed this benchmark using 69,721 queries and three distinct relevance judgment sets to create a robust leaderboard for information retrieval models.
Why it matters: As AI search models replace traditional keyword indexing, standardized benchmarks are moving from small, static datasets to massive, real-world web scale data.
Decoder
- Retrieval model: An AI system tasked with finding the most relevant documents or passages from a large corpus in response to a user query.
Original article
Q2D-Web is a large-scale benchmark and leaderboard for evaluating retrieval models on web search, encompassing 190 million documents and 69,721 queries in ten languages. It uses three separate sets of relevance judgments to minimize bias and improve reliability, assisting retrievers in distinguishing relevant data from distractors. Evaluation submissions are processed through a subsampling method to reduce resource costs while maintaining model ranking integrity.
Explore all 54 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi