DEVOURED
What data agent benchmarks do and don't tell us
AI Council attendee Jason Ganz notes the collapsing divide between data and AI infrastructure, highlighting that agent benchmarks are evolving to emphasize statefulness and rich, cross-system context over isolated tasks.
What: Jason Ganz, reporting from AI Council (formerly Data Council) on May 17, 2026, observes that most companies now position themselves as AI infrastructure providers, focusing on context retrieval, orchestration, or inference. He notes that new systems like LanceDB are built specifically for LLM workloads. Benchmarking efforts, including dbt Semantic Layer tests and ADE-bench, show agents perform better with well-specified tasks and extensive context from systems like GitHub and Slack, with Izzy Miller's 90-day business simulation benchmark further demonstrating the value of stateful learning.
Why it matters: The convergence of data and AI infrastructure, coupled with the shift towards stateful, context-rich agent benchmarks, indicates a maturation of the AI agent ecosystem, moving from isolated experiments to integrated, long-running, and truly intelligent systems that operate across an organization's entire knowledge base.
Takeaway: If you are evaluating AI agents, prioritize benchmarks and solutions that emphasize stateful learning and the integration of diverse organizational context (e.g., dbt, GitHub, Slack) for more realistic performance assessments.
Deep dive
- The AI Council (formerly Data Council) conference in 2026 showed a clear convergence of data and AI infrastructure.
- Many companies are now positioning as AI infrastructure providers, focusing on context providers, workflow orchestrators, or compute providers.
- New databases like LanceDB are being designed specifically for AI-native, multimodal workloads, handling blob reads and embeddings.
- Benchmarking for AI agents is evolving beyond simple question-answering, with efforts like dbt Semantic Layer tests showing performance improvements for LLM-generated queries.
- ADE-bench evaluates agents' ability to build data pipelines, moving beyond just answering questions.
- Opeyemi Fabiyi's research identifies specific activities that uplift agent performance with dbt projects.
- Izzy Miller's "90-day simulation" benchmark emphasizes statefulness and learning from mistakes over time, a critical aspect of real-world agent operation.
- Agents perform significantly better when given access to rich, cross-system context from sources like dbt projects, GitHub, Slack, and Notion, which is often underestimated in sandboxed benchmarks.
- A major upcoming constraint and optimization area is token and compute efficiency, as agent usage costs are already high.
- The article concludes that data agents are improving, and the industry needs better mechanisms to track performance with organizational context and memory, alongside a focus on efficient agent design.
Decoder
- AI Council (formerly Data Council): A prominent conference for the data and AI community, covering trends and developments in both fields.
- AI infrastructure company: A company that provides foundational technologies or services (e.g., context retrieval, orchestration, inference compute) to support the development and deployment of AI agents and applications.
- LanceDB: An AI-native multimodal lakehouse database designed from scratch for LLM-shaped workloads, focusing on blob reads and embeddings.
- dbt Semantic Layer: A component of dbt (data build tool) that defines consistent business metrics and semantics, improving the reliability of LLM-generated queries.
- ADE-bench: A benchmark designed to measure the ability of coding agents to build data pipelines, extending beyond simple data question answering.
- Statefulness: In the context of AI agents, the ability of an agent to retain and use information, memories, and learning from past interactions or tasks over extended periods, rather than starting fresh with each new task.
Original article
Full article content is not available for inline reading.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi