DEVOURED
How we built a lab to evaluate data agents
Hex developed Shoebox, an internal "lab bench" for data agent evaluation, and created Shorelane Commerce, a realistic fake business with messy data, to benchmark agents against real-world analytical challenges.
What: Hex created Shoebox for evaluating data agents, comparing "candidate" runs against "production baselines" across various parameters like prompts, models, and memory. They also built Shorelane Commerce, a synthetic B2B2C business founded in 2019 with $129M yearly revenue and six years of messy, realistic data (e.g., lost customer IDs, unmerged acquisitions, renaming channels) to provide a challenging environment for agents.
Why it matters: This initiative highlights the severe limitations of existing public benchmarks for data agents, which often use "demo-shaped" data, and underscores the necessity of building complex, real-world-simulating environments to genuinely assess AI agent performance in enterprise analytics. It shows that effective agent development requires moving beyond simple text-to-SQL tasks.
Deep dive
- Shoebox: Hex's internal evaluation infrastructure designed as a lab bench for agent observability and evaluation. It supports ad-hoc and scheduled evaluations, pairwise comparisons, and experimental treatments.
- Pairwise experiments: Shoebox is designed to compare a "candidate" agent run against a "baseline" run, allowing for objective assessment of improvements.
- Hybrid workflow: Engineers can run candidate evaluations locally and compare them against shared, consistent remote production baselines.
- Flexible rubrics: Eval sets come with preconfigured rubrics and ground truths, but users can create custom deterministic, LLM-judged, or hybrid rubrics, including run-scoped "hypothesis objective" rubrics.
- Shorelane Commerce: A synthetic B2B2C office-supplies platform with realistic, messy data designed to mimic real-world data warehouses, featuring data debt, multiple revenue streams, and conflicting data definitions.
- Realistic data challenges: Shorelane Commerce includes issues like lost customer IDs, unmerged acquired company data, renamed sales channels without backfilling, and multiple plausible "revenue" columns, forcing agents to handle ambiguity and data debt.
- Context over models: Hex's experience indicates that agent performance is more a function of the rich context stores they access (like workspace guides and semantic models in Shorelane) than just their prompts or underlying models.
Original article
Full article content is not available for inline reading.
Explore all 68 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi