LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search

Back to Devoured - August 06, 2026

DEVOURED

LLMs for Relevance: Automating High-Quality Product Relevance Labeling in Flipkart Search

Flipkart automated product relevance labeling by training a model on reasoning traces, achieving 30% lower costs while matching manual NDCG scores within 1%.

What: Flipkart uses an 8B parameter reward model with GRPO (Group Relative Policy Optimization) to evaluate explanation quality in product search relevance. This approach significantly reduces the cost of maintaining high-quality search labels.

Why it matters: This signals a shift from using LLMs just for classification to using them as automated arbiters of search quality via reasoning traces.

Decoder

NDCG (Normalized Discounted Cumulative Gain): A metric for measuring the quality of ranking algorithms, prioritizing relevant items at the top of results.
GRPO (Group Relative Policy Optimization): A reinforcement learning method where a policy is optimized by comparing multiple outputs from the same prompt to find the most accurate one.

Original article

Flipkart's Product Analyser trains a relevance labeling model in two stages: SFT on millions of balanced query-product examples with generated reasoning traces, then GRPO alignment using an 8B reward model that scores explanation quality alongside label correctness. The automated NDCG report it produces differs from the manual one by under 1%, at 30% lower cost.

Explore all 58 articles in today's digest