DEVOURED
Text-to-SQL Models Make Choices Without Saying So
Text-to-SQL models often silently resolve ambiguous user requests by picking one interpretation without disclosure, leading to inaccurate results.
What: Researchers Yichen Xu, Shipei Lin, and Chenhao Ma introduced SpecBridge, a tool that forces models to output multiple interpretations for ambiguous SQL queries, increasing coverage by 29-63% without hurting clear requests.
Why it matters: This reveals a critical failure mode in current AI agents: silently prioritizing a single path when faced with ambiguity, which creates a 'hidden' inaccuracy that standard benchmarks fail to measure.
Deep dive
- Text-to-SQL models frequently make arbitrary choices on ambiguous natural language queries (e.g., counting rows versus distinct entities).
- Current benchmarks (like BIRD) are insufficient because they only grade against a single reference answer.
- SpecBridge tests for ambiguity by generating three queries—one neutral and two specific—to verify if models can surface both interpretations.
- The research found that models are capable of computing multiple interpretations but fail to report them in 15-58% of cases.
- Systems can improve accuracy by using a two-step process: first detect if a request is ambiguous, then execute multiple interpretations in parallel.
- This method requires fewer user turns than standard clarifying questions.
Decoder
- Text-to-SQL: The task of using an LLM to convert natural language queries into executable SQL commands for a database.
- SpecBridge: A framework introduced in the paper to evaluate how models handle ambiguous SQL generation tasks.
- BIRD Benchmark: A large-scale, cross-domain dataset used to evaluate the accuracy of text-to-SQL models against real-world database complexities.
Original article
Title: Text-to-SQL Models Make Choices Without Saying So
A text-to-SQL answer can run, return a plausible number and still compute something other than what the analyst meant. This can happen when the request leaves a calculation choice open, such as whether to count rows or distinct entities, and the model settles it without saying so. Standard benchmarks cannot detect this, because they score each question against a single reference interpretation. We introduce SpecBridge, which tests such a choice by varying it alone: it poses three otherwise identical requests, one leaving the choice open and one per interpretation, and scores each answer by running it. The databases are real, and the choices are ones that made models fail on existing benchmarks. On 204 hand-validated tests, four language models usually compute each interpretation when asked for it, yet for the open request return both in only 15-58% of the tests where they can, often without mentioning the choice. The gap lies in reporting, not in computing. Benchmarks even penalize the alternative: on BIRD questions where such a choice changes the answer, over two in five of the answers marked wrong are right under the other interpretation. Because the gap lies in reporting, a system can narrow it without a stronger model: it first decides whether a request leaves a choice open, then answers each interpretation in its own call. On our tests and on AMBROSIA, an existing benchmark of ambiguous questions, this approach raises the share of open requests that get every interpretation by 29-63 points over the single answer that production assistants give, without lowering precision on clear requests. It also needs fewer user turns than asking a clarifying question.
Explore all 58 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi