Introducing DataBench

A frontier benchmark for complex data work and analytical reasoning

Here are two prompts you might give to an agent:

Construct a counterexample to general (non-planar) case of Dinitz Garg Goemans conjecture. You should do a breakthrough and find a structured counterexample.
Was the Midwest acquisition v2 campaign worth the spend? Ad platform numbers look like it drove 837 conversions…

Which seems more likely to fail?

Well, a mathematician recently used the first prompt with GPT-5.6 to make a correct and significant breakthrough on a famous math problem that’s been stumping experts for 30 years.

We ran the second, trivial looking prompt in a warehouse environment and got a confidently incorrect answer that claimed the campaign wasn’t worth it, when really the data necessary to make that call doesn’t exist.

So what gives? Why is data so hard!? Do I really need Fable 5 Max to do my marketing performance roundups? Can’t Sonnet do it? In this strange era of jagged machine intelligence, how do agents perform on the agentic analytics tasks that we care most about at Hex?

To find out, we created a new frontier benchmark for agentic analytics: DataBench.

You can skip straight to the full results if you want by clicking here — or read through the next section to hear why we made Yet Another Benchmark.

The cursed domain

I have always maintained that data analytics is a uniquely difficult domain for agents to operate in. I’ve previously written at length about what makes agentic analytics so challenging:

Easy questions look hard. Hard questions look easy. Many questions are impossible to answer; to even try is to fail. Bugs are usually silent and subtle. Innocuous assumptions (LLM’s favorite!) make or break analyses. There are no linters, no test suite, no formalization language. There is almost no realistic public data to train on or build environments from, and there is a surplus of unrealistic tutorial-slop jamming up the pretrain. Everyone’s data warehouse is out of distribution. For every right answer, there are ten plausible but subtly incorrect wrong answers, and no way to verify or validate the result.

But there has been a tension between this claim and what the leading industry benchmarks seem to indicate. Sonnet 4.5 sits at a comfortable 90% on Spider 2.0. Claude Haiku 4.5 can get 89% on DABstep. Recent attempts at the newer Data Agent Bench are all coming in around the 85%+ mark.

So is analytics actually totally solved?! We don’t even need frontier models? Alas, not yet.

The problem is that these benchmarks don’t really test for the kinds of agentic analytics tasks that we see customers running in Hex. Instead, they consist of what I like to call “overspecified pub trivia”.

These can be impressive to demo, and it is great that models are getting good at this stuff, but unfortunately this is not how normal people prompt their agents! These are more akin to “feats of strength” than realistic usage.

These are very different tasks! They are vague and directional. The user often isn’t sure exactly what they want. They are almost never asking for a single number, they want a recommendation, a gut check, or a foothold for where to look next. Often the right answer is “I’m not sure we can answer that honestly” (spoiler: this is where models perform worst, they do not like giving up).

Even in a well-modeled semantically rich environment, these kinds of tasks require traversing broad swaths of the warehouse and making hundreds of decisions about definitions, context semantics, data quality, user intent, and analytical best practices.

We needed to understand how models perform on the things we really care about at Hex! So we built a more representative, realistic benchmark that looks like the work people are actually doing: DataBench.

Performance on DataBench

DataBench v1 covers 100 realistic analytical tasks split across Q&A and Open-Ended prompts. Everything runs in the Hex workspace of a synthetic business called Shorelane Commerce, and is executed & judged using the native Evals functionality.

It’s still a v1 and there is a lot of room for improvement, but we feel it is the first benchmark for agentic analytics that truly targets the kind of real work people want agents to be able to do in this domain.

The most actionable insights:

Opus 5 is capable of greatness but behaves very oddly at higher effort levels.
Claude Fable 5 is the only model where high effort doesn’t backfire.
GPT 5.6 Sol is often “good enough" at 1/2 the cost.
GPT-5.6 Luna is absurd bang for your buck.
Despite the strange high-effort behavior, Opus 5 is a meaningful upgrade on Opus 4.8— forget what the haters on X say.
Sonnet 5 is a bit of a confusing model and probably rarely the right choice.

Where the models shine

The floor is higher than we thought

The most striking thing about this chart is where the performance floor is. No model/effort pair scores worse than 50%, despite none of the DataBench tasks intentionally being “easy”. Tasks exist across an array of difficulties, but there are tricky nuances to even the simplest ones.

Models can take in massive amounts of tokens and consider them all at once. This lets them notice analytical details across complex queries that almost seem superhuman! Looking at 250,000 tokens of query results, what’s obvious to an agent is far from what’s obvious to a human analyst, and this is where the model’s best performances consistently lie.

The early Pareto frontier is steep

“Pareto efficiency frontier" is just a fancy way of saying “for a given budget, what’s the best score my money can buy?” It follows the dotted line at the top-left edge of all our plots.

We were surprised to see the incredibly strong relative performance of GPT-5.6 Luna, which forms the entire pre-elbow section of the Pareto efficiency frontier! At xHigh effort, it achieves near-Sol performance at ~1/14th the cost.

Messy data is easy when intent is clear

About half the difficulty in DataBench comes from having to accomplish relatively clear tasks in a complicated and messy data environment— multiple definitions of the same metric, fragmented definitions that live across difficult-to-join tables, stale and half-deleted datasets. We assumed this would be a major source of failure.

It’s actually mostly fine? It turns out that this sort of environment is not very challenging for frontier models when the intent and guardrails of a task are relatively clear.

Where models struggle

So where do things break down? We broke the scores out by task type and there’s a clear pattern: models are best at gathering evidence (75% on Q&A tasks), worse at open-ended delegated decisions (66%), and worst on the specific “trap” failure modes we’ve baked into DataBench where there is an obvious and plausible but wrong easy answer and success requires going deeper (54%).

What this boils down to is judgment. The smallest models hang surprisingly close to the frontier on Q&A, but fall way behind on the traps that require intuition and reasoning.

Making the right calls

Agents can run all the numbers "correctly" under very specific assumptions of what “correct” means, but still draw the wrong second or third-order conclusion. Unlike the collections case above, this doesn't generally come from trusting bad evidence— it comes from manufacturing certainty.

You should still be careful outsourcing complex decisions like this to agents.

A great way to mitigate this is to be curious and follow up with agents! Poke on things, pressure test, keep your brain turned on! Please keep your brain turned on. Rest assured we will let you know as soon as you are able to turn your brain off.

Difficulty catching mistakes

Humans are remarkably good at a kind of applied suspicion that models do not display. “Oh, that number doesn’t look like what I expected it to!” is not something we frequently see models say. A human analyst working on these problems says that A LOT. This spidey-sense informs the way humans catch mistakes, reorient, and adjust their confidence levels.

Frontier models do not like to throw in the towel

When we compare DataBench’s result curves to coding benchmarks like CursorBench, we see much less uniform improvement from test-time scaling and model size. Unlike these benchmarks, we see regressions at high efforts, especially for Opus 5. What gives?

This is what makes agentic analytics so uniquely challenging: there is no test suite and no formal or even informal verifiability. On a complex SWE task, it is hard to imagine spending more time and effort to get a worse outcome. Maybe you plateau, but you don’t regress, and the coding benchmarks show this very clearly.

Analytics does not work this way. Knowing when you have the right answer is a matter of judgment and vibes more than anything else.

Task details

There are 100 tasks in DataBench v1 and every single one of them is carefully crafted to not just be a “translate these requirements into SQL” prompt. Instead, we send ambiguous questions with open ended possibilities for answers, and place the agent in a warehouse filled with golden semantically modeled data as well as dangerous (but sometimes necessary) raw tables.

Tasks

There are two types of tasks in DataBench v1: Q&A and open-ended. Ten of these are also the “signature traps” you met above.

Q&A tasks are straightforward but not over-specified analytical queries.

Open-ended tasks require the agent to go a step further, either creating an artifact or making a call for the user instead of just providing a direct quantitative answer to a question.

Rubrics

Every task is evaluated by an LLM judge that has access to the thread, the artifacts it creates, a ground-truth “expected” dataset where relevant, and a comprehensive rubric for evaluation.

Environment

Everything runs in the Shorelane environment, the same workspace and warehouse we use for internal development and evals.

What comes next

We’ll be making continuous updates to DataBench and publishing new versions regularly as well as adding new models. Immediate future improvements involve increasing the number and complexity of “artifact” based tasks to better evaluate things like Generative Data Apps.

It is helpful for our internal development signal to keep DataBench private so the tasks don’t get trained on, but we do have plans to open-source the Shorelane analytical environment.