Can data agents do real analysis? (51 minute podcast)
Current data agents struggle with complex analytics, hitting only 61% accuracy on realistic benchmarks that involve multi-database joins and messy unstructured data.
Deep dive
- Current benchmarks often overfit to clean, stylized tasks and fail to capture real-world analytical complexity.
- Data agents struggle significantly when required to perform joins across multiple disparate databases.
- Agents currently lack an equivalent of human pre-attentive processing (e.g., visual charts) to quickly identify interesting data points.
- Using a semantic layer significantly increases agent performance by grounding query generation in defined business metrics.
- LLM judges suffer from 'criteria drift', where the definition of success evolves as developers review more output, requiring manual oversight.
- AI-powered SQL queries are currently inefficiently implemented as row-by-row LLM calls; custom query planners are needed to optimize costs.
Decoder
- Semantic layer: An abstraction layer between raw data and analytics tools that translates database schemas into human-readable business definitions (e.g., "Revenue" instead of
select sum(price) from table_a). - Pass@1: A performance metric representing the probability that an agent provides the correct answer on its first attempt.
- Pre-attentive processing: The subconscious accumulation of information from the environment, which in human data analysis is often driven by visual patterns in charts.
Original article
Can data agents do real analysis?
Carnegie Mellon Professor Shreya Shankar on why data agents fall short at real analytics work, what a semantic layer changes, and how to evaluate LLM outputs you can trust.
Can data agents do real analysis? Shreya Shankar’s answer is that there is still a big gap. The best agent in her Data Agent Bench passed 61% of queries on the first try, and agents did significantly better when they had a semantic layer.
Shreya Shankar did her PhD in databases at UC Berkeley and joins Carnegie Mellon University as an assistant professor in 2027, where she will be part of the CMU Database Group. Her research sits where databases, AI agents, and human-computer interaction meet. She is the creator of DocETL, a system for LLM-powered data processing, and a co-author of the Data Agent Bench (DAB), a benchmark that tests whether AI agents can do the work of a junior data analyst. Her paper “Who Validates the Validators?” examines how developers build and trust LLM judges.
She joins Tristan Handy for a conversation about what happens when you ask AI agents to do analytics in a realistic environment: multiple databases, messy data, tribal knowledge, and free-text columns. The best agent in her benchmark passed 61% of queries on the first try, and agents that sat on top of a semantic layer did significantly better. They also get into evals, including why you can’t know what you want from an LLM judge until you have looked at a lot of outputs, and why there is still no equivalent of df.summary() for agent outputs.
Along the way: why data agents have no counterpart to the visual shortcut humans get from a chart, why humans are “actually awful at writing SQL” and SQL will stick around anyway, why agents want to select * from two databases and join them in pandas, why her defense of AI evals came down to what actually moves the quality and cost frontier, and what she plans to do about AI SQL queries that cost far too much.
Three ideas from the episode
1. Data agents have no shortcut like the one visualization gives humans. Shreya describes an “inner loop” of tools like databases, which answer a question quickly once you know it, and an “outer loop” of tools like business intelligence, which help people figure out the right question in the first place. Charts work because people can see what’s interesting without spending reasoning effort. Agents have no analog of that yet, and Shreya’s open research question is what you can put in the prompt so an agent spends fewer reasoning tokens to get to the right answer.
2. Realistic benchmarks show a bigger gap than the clean ones do, and a semantic layer closes part of it. Existing benchmarks are either text-to-SQL, where the agent only writes a query, or table question answering, where the whole table goes into context. Shreya’s team built one with multiple databases, domain knowledge, and unstructured text to transform. The best agent got 61% pass@1. Giving agents a semantic layer significantly raised accuracy, which Shreya says speaks to the importance of the semantic layer, and of humans who understand what is in it.
3. You can’t judge LLM outputs well without looking at them first. In the “Who Validates the Validators?” study, developers’ criteria changed as they saw more outputs. Shreya calls this criteria drift. Two years later she says it still holds: to scale up LLM judges you have to read enough samples yourself to know what your rubric should be, and today that mostly means reading rows in a spreadsheet.
Everything referenced in this episode
- Shreya Shankar's website: https://www.sh-reya.com
- Data Agent Bench (DAB), "Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents": https://arxiv.org/abs/2603.20576
- DocETL: https://www.docetl.org
- "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences": https://arxiv.org/abs/2404.12272
- ChainForge: https://chainforge.ai/
- "In Defense of AI Evals, for Everyone": https://www.sh-reya.com/blog/in-defense-ai-evals/
Key takeaways
Why did you put a note for AI agents on your website?
Shreya Shankar: I got a new job as a professor of computer science a couple of months ago, and all of a sudden I started getting an uptick in email requests. Even as a PhD student, people would email to say they wanted to do research with me. Once AI agents became popular it got easy to send a four or five paragraph email where every paragraph is a different paper of mine, and it’s all AI generated.
I was tired of reading those, so I put a note on my website. It took a couple of days, but within the first week the number of emails went down significantly. I used to get about five a day, and now I get about three a week. I’ve only gotten one email with the phrase the note asks for, so I don’t think I’ve made it a good prompt injection. My guess is that the agents mass-emailing professors are just not writing to me anymore, and I’m very happy with that.
I actually tell people now, just don’t use AI to write emails and you will get noticed. It’s never been easier to get a job or get noticed, because all you have to do is not use AI.
How does a machine learning engineer end up with a database PhD?
Shreya Shankar: I did my undergrad at Stanford during the deep learning boom, and did a couple of internships at Google Brain. I loved research, but I didn’t know what I would want to do a PhD in, so I went to industry. I was the first ML engineer at a data analytics company nobody has heard of, and what I found was that none of my problems were machine learning or AI problems. They were data engineering problems.
How do you do the data prep to harness machine learning at scale? That got me interested in databases and the idea that we should be able to build declarative systems and have the database engine optimize the queries, so data engineers don’t sit and hand-tune every one. I thought I wanted an AI PhD, but it turns out nobody in AI is interested in these problems, and everybody in databases is.
Should data agents interact with data the way humans do?
Tristan Handy: Analysts have always done two things: answer questions somebody else generates, and, as they get more senior, find what’s interesting. Humans have a very limited working memory, so it seems like agents should be better than we are at looking at a big blob of data and telling you what’s interesting. And yet agents today interact with databases mostly the way humans do. We say, you’re good at SQL, go write some SQL.
Shreya Shankar: My framing is that there’s an inner loop of data tools, like databases, that are responsible for executing a question and getting the right answer quickly. Then there’s an outer loop of tools like business intelligence, which help the user figure out the right question, because they don’t know it in the first place.
One of the prime innovations there is visualization. It taps into our pre-attentive processing. We have a very strong visual cortex, so we can look at a plot and, without tapping into our reasoning tokens, see what the interesting question is. All of the good questions come from visualization, or at least it’s the seed. There’s no analog of this right now for data agents.
What is a data agent, and what did the benchmark test?
Shreya Shankar: A data agent is some AI or entity interacting in the traditional business setting. There are many databases available, it interacts the way a human data analyst would, and it’s tasked with answering a question a human analyst would try to answer. I consider my data agent a junior analyst who uses all the same tools and has to work with all the same data I have. We scoped it to questions with a verifiable answer, like computing sales, or figuring out which customers to prioritize based on who wasn’t repeatedly buying our product.
Can an LLM do relational algebra in context?
Shreya Shankar: I’m sure Fable can take in a whole table and come up with something interesting. I just think the majority of queries are relational algebra in some form, and so you should use a database to do it.
What makes an analytics environment realistic?
Shreya Shankar: We built our own benchmark, and it took a long time to figure out how. We talked to a lot of people and came up with four conditions analysts actually work under. First, you have multiple databases or data warehouses in the environment. Second, an analyst needs to push computation down to the individual databases as much as possible. Third, domain expertise has to be encoded somewhere. Fourth, you often need some unstructured data transformation.
How did the agents actually perform?
Shreya Shankar: Right now the best agent is at about 61% pass@1 over our queries. If you look at our leaderboard you’ll see numbers in the 80s, but a lot of people are overfitting to the task and encoding exactly how to answer in the prompt. As a benchmark admin it’s hard to tell whether someone is cheating. Even with prompt-engineered Fable, people are getting around 85%. That’s pretty good, but I think there’s still a big gap.
Do data agents work better with a semantic layer?
Shreya Shankar: It’s consistent. We have different variants of the benchmark, including one where we provide a semantic layer, and that significantly increases accuracy. It speaks to the importance of having a good semantic layer. OpenAI and Anthropic both have blog posts on how they’ve built their own semantic layers, and it’s a P0 for them to get their agents to do data analysis.
What did “Who Validates the Validators?” find about LLM judges?
Shreya Shankar: I thought it would be a flashy paper saying these ideas apply to LLM judges and it all works. It’s not going to work that simply, because human preference depends on what you’re seeing. People don’t know what they want until they see the outputs, and they have to see enough of them to develop a theory of how the model works before they feel confident. We call this criteria drift.
Why defend AI evals?
Shreya Shankar: My takeaway is that the thing that pushes the Pareto frontier of quality and cost the most is a new model. Prompt engineering and harness improvements are great, but in the hands of the end user they’re limited.
How should data practitioners start doing evals?
Shreya Shankar: The concept of looking at your data is drilled into you as a data analyst or data scientist, and it’s paramount now. The new part, which is tricky, is that exploratory data analysis doesn’t cleanly map from structured data analytics to agents operating over unstructured data. There’s no equivalent of df.summary(). You’re going to have to write the code yourself, because there are no libraries for it. But look at your data, and try to do it in a principled way.
What are you working on next at CMU?
Shreya Shankar: One big area is AI SQL. Snowflake, Google BigQuery, and Databricks have shipped a variant of SQL that lets you write AI operators in your queries. It’s bringing the database companies a lot of revenue and everybody is thrilled, except for people who accidentally spend hundreds of thousands of dollars on their queries. I think it’s way too expensive, and we haven’t even scratched the surface of how to do query optimization here.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi