The most useful number in analytics is a subtraction. Give a frontier-model agent the benchmark that made the field declare text-to-SQL solved and it scores 91.2%. Rebuild that same benchmark so the schemas look like a real company’s — Spider 2.0, the enterprise-realistic successor, averaging about 800 columns, with workflow tasks whose answers can exceed a hundred lines of SQL — and the paper’s best agent managed 21.3% at launch. Seventy points, opened purely by making the problem look like work.
So whether an AI agent can be trusted with your database depends entirely on which database you mean. Ask one to query a tidy, well-named, classroom-scale schema and the benchmarks say it will succeed nine times out of ten. Point it at a real enterprise warehouse — hundreds of tables, thousands of cryptic columns, metrics whose definitions live in someone’s head — and the best published end-to-end systems still fail most of the time. Both facts are true simultaneously, both are measured, and the distance between them is where every “chat with your data” product lives. I work in this exact space, so consider this my attempt to put the public evidence in one place.
Spider (2018) — the Yale cross-domain benchmark — asked models to write SQL against databases they had never seen. The best system at launch managed roughly 10% exact-match accuracy. By late 2023 the top of the leaderboard had passed 91% execution accuracy, and the research community declared the benchmark effectively solved.
BIRD (2023) — built deliberately messy, with real database contents, not just schemas — reported a humbling launch statistic: GPT-4 scored 54.9% execution accuracy where human data engineers scored 92.96%. As of the leaderboard this week, the best published systems have climbed to roughly 82% — a real narrowing, and still eleven points under the humans, three years on.
Spider 2.0 (2024) is the seventy-point cliff I opened with, and nothing about it is exotic: it is what the same task looks like once the schemas, the column names and the multi-step workflows resemble the ones a company actually runs on.
BEAVER (2024–26) — built from real private enterprise warehouses and their actual query logs, averaging over a hundred tables and ~869 columns per database — is the current floor: frontier models score around 11% end-to-end as of the latest published evaluation. Give the models oracle-provided task decompositions and accuracy roughly triples — which locates the bottleneck precisely: not writing SQL, but understanding what the question means against this schema.
One nuance the leaderboards add: on some Spider 2.0 variants, agentic scaffolds have climbed spectacularly since the paper — the Snowflake-hosted subset’s top entry now posts 96.7%, roughly eighteen months of harness engineering doing what raw model scaling did not. But the subset that requires navigating an actual dbt code project still tops out around 65%, and BEAVER’s real-warehouse floor remains stubborn. Scaffolding moves the number; realism moves it back.
The failure decomposition across these benchmarks is consistent, and none of it is about SQL syntax. Enterprise schemas are enormous and cryptically named, so the model retrieves the wrong tables before it writes a single clause. Business questions are ambiguous — “revenue last quarter” has three defensible definitions in most companies, a problem I have written about before, and the model resolves the ambiguity by silently picking one. Real data is dirty in ways only veterans know (“status 7 means cancelled, but only before the 2021 migration”). And enterprise questions are workflows, not queries — multi-step, multi-system, dependent on context no schema encodes. My favorite detail from the literature: researchers were publishing taxonomies of human SQL errors back in 1997. Confidently wrong queries predate the model by decades. What changed is how fast they get produced now.
The clearest evidence from 2026 concerns the semantic layer — giving the model governed definitions of metrics and entities instead of raw schema. The caveat first, because it matters: essentially every published semantic-layer accuracy study comes from a vendor that sells one. With that stated: dbt Labs’ 2026 benchmark is the strongest of the lot precisely because its dataset and code are open-sourced — raw text-to-SQL on a well-modeled project scored 84–90% across the two frontier models tested, and the semantic-layer path scored 98–100%. Snowflake’s engineering team reports a lift from 57% to 78% on BIRD from adding a semantic model. Vendor numbers, both — but they agree in direction with the independent BEAVER finding that decomposition and grounding, not generation, are the bottleneck.
There is one more finding in the dbt writeup that deserves to outlive the benchmark, because it reframes the whole question. The two approaches fail differently. The semantic-layer system’s failures were mostly refusals — “I can’t answer that.” Raw text-to-SQL’s failures were confident, plausible, wrong numbers. Readers of my decision-intelligence essay will recognize why that asymmetry is everything. A refusal costs you a query. A confident wrong number, delivered fluently to an executive, costs you the program — because trust, once spent, does not refill at benchmark speed.
Which is why “can AI be trusted with the database” is the wrong question — un-scoped trust isn’t what serious systems ask for. The right question is the one aerospace asks of its autonomy, and the answer pattern is the same one running through this whole series: trust is manufactured, scope by scope. Bound the schema the agent sees. Govern the definitions it reasons over, so ambiguity is resolved by policy rather than by the model’s mood. Verify by execution — run the query, check the shape of the answer — before a number reaches a human. Prefer refusal to confabulation at the edge of scope. And measure against your warehouse, not Spider — because the single most replicated finding in this literature is that the distance between the benchmark and your production schema is where the accuracy goes to die.
Under those conditions, the 2026 evidence says yes: bounded, grounded, execution-verified agents are already answering most well-posed questions correctly, and improving fast. Without them, the evidence also says yes — to an agent that will answer everything, confidently, including the questions it got wrong.
References