The benchmark grading every text-to-SQL model has wrong answers in its key

BIRD and Spider are the exams of the text-to-SQL world. Every model you've seen ranked — GPT, Claude, Gemini, every fine-tuned SQLCoder variant — gets its accuracy number by comparing its output against these benchmarks' gold queries: SQL written by experts, reviewed, and published as ground truth.

We pointed a deterministic semantic checker at the ground truth itself. It took about two seconds, and one of the answers in the key is provably wrong by a factor of eight.

That finding is the hook. The reason to keep reading is what it implies: the way this entire field measures itself has a blind spot — and it's the same blind spot that ships wrong numbers to enterprise dashboards every day.

Layer 1 — Gold isn't always gold

The method (no AI, no labeling)

sqlsure validates SQL against declared facts — what one row means, which joins multiply rows, what's safe to sum. Benchmarks conveniently publish those facts: every BIRD and Spider database ships its primary and foreign keys. So the audit is mechanical: build the rulebook from each database's own declarations, parse all 2,568 gold queries (1,034 Spider dev + 1,534 BIRD dev), flag joins that contradict the declared facts, review every flag by hand — and, for BIRD, by executing the shipped databases.

The 8× answer

BIRD dev question #571 asks: "For the user No.24, how many times is the number of his/her posts compared to his/her votes?" We executed the benchmark's own database: user 24 has 3 posts and 8 votes. The correct answer is 3 ÷ 8 = 0.375. The gold query returns 3.0 — its join creates a 3 × 8 cartesian product per user, and the inflated count divided by the distinct count algebraically collapses to… the post count. Off by exactly the fan-out factor.

-- BIRD gold (returns 3.0):
SELECT CAST(COUNT(T2.Id) AS REAL) / COUNT(DISTINCT T1.Id)
FROM votes T1 JOIN posts T2 ON T1.UserId = T2.OwnerUserId
WHERE T1.UserId = 24

-- correct (returns 0.375):
WITH v AS (SELECT COUNT(*) c FROM votes WHERE UserId = 24),
     p AS (SELECT COUNT(*) c FROM posts WHERE OwnerUserId = 24)
SELECT CAST(p.c AS REAL) / v.c FROM p, v

We also found a schema-level defect — european_football_2 declares 29 foreign keys on its Match table but omits the league link its own gold answers use 13 times. And the training set is worse: our follow-up audit of all 9,428 BIRD train queries found 8.2% of joins unbacked by any declared relationship, including eight databases that declare zero foreign keys at all.

After our audit, we found the expert-corrected BIRD dataset from the VLDB'26 annotation-errors project: 10 of our 15 dev flags were independently identified by their human review, and their expert fix for #571 computes exactly our 0.375.

Layer 2 — Execution accuracy is not ground truth

Here's the part that matters more than any single wrong answer. The field's standard metric — execution accuracy — asks: does the generated query's result match the gold query's result?

To be fair to it: that design elegantly solves a real problem. SELECT SUM(sales) and an equivalent subquery formulation are different SQL with the same answer — comparing results instead of text correctly accepts both. Equivalent SQL is the failure mode execution accuracy was built for, and it handles it well.

But it has an unexaminable assumption: that the gold result is right. Follow the chain when it isn't:

question → gold SQL (wrong) → gold result (wrong) → execution accuracy
                                                      rewards matching the wrong result
                                                      penalizes the correct answer

A model that answers #571 correctly — 0.375 — is marked wrong. A model that makes the same fan-out mistake as the annotator is marked right. At scale, this doesn't just add noise: correcting annotation errors has been shown to materially change reported performance and even reorder model rankings. Companies choose AI systems off these leaderboards. Procurement decisions are partly downstream of annotation bugs.

Layer 3 — Enterprise SQL is harder than any benchmark

Suppose every benchmark answer were perfect. The gap between "passes the benchmark" and "trustworthy in production" would still be wide, because enterprise analytics adds semantics no execution check can see: joins that fan out and silently double-count, non-additive measures (you can't sum averages), semi-additive measures (Monday's bed count plus Tuesday's isn't two days of beds), canonical metric definitions, row-level security, fiscal calendars, hidden business filters. A query can be syntactically valid, execute cleanly, and match another query's result while violating every one of these. Correct SQL syntax ≠ correct analytics — and the experts who wrote BIRD's answer key just demonstrated that even careful humans fall into exactly these traps.

Layer 4 — Constraint-aware evaluation

There's a complementary way to judge SQL: not "does the result match gold?" but "does the query respect the declared semantics of the data?" — grain, join cardinality, additivity, metric definitions, column policy. Facts that teams already declare in dbt tests, PK/FK constraints, and semantic layers.

The two axes are complements, not rivals: execution accuracy checks outcome against a reference; constraint validation checks reasoning against declared truth — needs no reference answer at all, which is exactly why it can audit the reference answers themselves. It's deterministic (same query, same verdict), and every rejection carries a fix a model can apply mechanically. We've published it as an open eval metric — a semantic pass rate to report alongside execution accuracy.

Why this matters if you're deploying AI on your data

• If you pick models off leaderboards: some of the ranking signal is annotation error. Ask vendors for semantic-validity numbers, not just execution accuracy.
• If you fine-tune on BIRD/Spider train: your model is learning from schemas where up to 8% of gold joins aren't backed by any declared relationship — including databases with no declared links at all.
• If your agents write SQL in production: there is no gold answer to compare against out there. Constraint validation is the only reference-free check that exists — which is why it belongs in the loop, not just in the eval.

The future of text-to-SQL isn't just better models

SQL generation has become good enough that evaluation is now the bottleneck. The next generation of benchmarks shouldn't only check whether results match a reference — they should verify whether queries respect grain, join cardinality, additivity, metric definitions, and policy.

pip install sqlsure