SQL Agent Schema Selection Layer Tested: Opus Loses to Sonnet, Voting Yields Zero Gain

Title

The author of the open-source project schemagate shared a set of rather counterintuitive benchmark results on Reddit: he ran benchmarks for the SQL Agent schema selection layer he developed on BIRD and Spider 2.0, and found that bigger models aren't necessarily better, while the seemingly safe voting mechanism yielded zero gain on the holdout set.

schemagate is an open-source retriever and MCP server for SQL Agents. Its role is clear: from the set of tables the caller has permission to read, pick the tables needed to answer the question, and validate the generated SQL against the same table list before execution. In other words, it plays two roles at once: "table selection" and "permission gatekeeper." The author emphasizes that all results are one model call per question, no voting, no Agent loops, scored by each benchmark's own evaluator.

Benchmark Results: Numbers and Gaps

On all 1,534 questions in the BIRD dev set, Claude Opus 5.5 achieved 77.1% execution accuracy; the hidden test set has been submitted and is awaiting results from the BIRD team. On the 135 SQLite questions in Spider 2.0-lite, Sonnet 5.5 reached 71.9%—the author notes that the best public Agent report he knows of on this subset reports 57.8% Pass@1, and uses a different model. Meanwhile, accuracy on the 205 BigQuery questions is only 45.9%, clearly more difficult; the Snowflake portion has not yet been run, so there is no leaderboard record yet.

These numbers form a clear but steep difficulty curve: performance on SQLite is approaching practical, while BigQuery still has a noticeable gap. The author also candidly acknowledges methodological limitations: a single run means roughly ±1–2 points of variance, the Spider 2.0 numbers cover only two of three parts, and the BIRD Mini-Dev row is self-reported until the leaderboard PR is merged.

BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation) is one of the most influential public benchmarks in the Text-to-SQL field today; its dev set contains 1,534 natural language questions across multiple databases, paired with human-annotated SQL and execution results, and is typically evaluated by Execution Accuracy—the proportion of generated SQL whose results match the gold answer when executed on the real database. Spider 2.0 is the second generation of the Spider series, specifically targeting enterprise-grade real data warehouse scenarios, covering BigQuery, Snowflake, SQLite, and other dialects, with longer questions and more complex schemas, and is considered significantly harder than the original Spider. BIRD and Spider 2.0 are often used together precisely because the former is closer to academic standard libraries and the latter to production environments and dialects; their scores cannot be directly compared.

Counterintuitive Finding 1: Opus Loses to Sonnet on Spider 2.0

With the same pipeline and the same 135 questions, Sonnet scored 71.9% while Opus scored only 67.4%. But on BIRD, Opus performed clearly better. This crossover result directly breaks the default assumption that "bigger model = better."

The author's advice is direct: don't assume the bigger model will win on your data—test it. The likely reason behind this phenomenon is that schema selection tasks place higher demands on a model's "instruction following and format stability" than on pure reasoning ability, and preference differences of different models on specific task distributions get amplified. For teams currently choosing a model, this means model selection must be based on your own data distribution, not leaderboard rankings.

"Schema selection / schema linking" is the upstream step in the Text-to-SQL pipeline: a database may contain hundreds or thousands of tables, while the context window and cost of a single prompt are limited, so a relevant subset must first be retrieved from all tables before being handed to the model to generate SQL. The accuracy of this step directly determines the ceiling—missing a table guarantees a wrong answer, while selecting too many tables crowds the context and introduces noise. So table selection is essentially a retrieval and ranking task, not a pure reasoning task, and is more sensitive to a model's instruction following, output format stability, and attention allocation over long lists. This also explains why different models' relative advantages in schema selection may not align with their rankings on general reasoning leaderboards.

Counterintuitive Finding 2: Voting Mechanism Yields Zero Gain on Holdout Set

The combination of multiple candidate queries plus majority voting plus a judge model showed about a +3 point improvement on half the questions, but on the held-out half the gain was exactly 0, at 5.5× the cost of a single call.

The author's relief is worth recording: "I'm glad I split the dataset before tuning." If only the tuning set had been examined, this approach would have been misjudged as effective. The failure of voting mechanisms in SQL generation points to a more general problem—when the errors of multiple candidates are correlated rather than independent, majority voting cannot offset the bias; it only amplifies cost.

Self-consistency voting and majority voting are common efficiency techniques in LLM reasoning: sample multiple reasoning paths or candidate answers for the same question and take the most frequent one, based on the theoretical premise that errors across paths are independent, so the correct answer is more likely to be "voted in." But when candidates are generated by the same model, same prompt, and same context, errors tend to be highly correlated—the model systematically errs at the same step, for example all misreading the meaning of the same table, in which case voting just repeats the same error. SQL generation is especially like this, because errors often stem from misunderstandings of the schema rather than random fluctuations, and such bias reproduces stably across multiple samples, so majority voting naturally cannot offset it.

Counterintuitive Finding 3: BigQuery Date-Sharded Tables Break the Selection Logic

One database had 366 tables named ga_sessions_20160801 through ga_sessions_20170801. When presented one by one to the model, they filled all 12 slots, and with random dates, the model simply refused to answer: "There is no table for January 2017."

The fix was to collapse each table family into a single wildcard table (ga_sessions_*, filtered via _TABLE_SUFFIX), and most refusals disappeared. The author warns: if your data warehouse has sharded tables, your retriever probably has this bug too. This is a trap that occurs frequently in real production environments but is easily overlooked in benchmarks.

Date-sharded tables are a common pattern in data warehouses like BigQuery: because early single tables had size and query cost limits, creating tables by day or month (e.g., ga_sessions_YYYYMMDD) became customary. Such tables appear in metadata as a large number of independent tables with identical structure and only different names, which is extremely unfriendly to retrievers—they flatten into the candidate list and crowd each other out, and the dates in table names are easily interpreted by the model as semantic information. BigQuery's wildcard table syntax (ga_sessions_* combined with the pseudo-column _TABLE_SUFFIX for filtering) lets the entire table family be queried as one logical table, reducing both metadata noise and scan cost. If a retriever does not perform table-family recognition and collapsing, it will systematically fail on databases dense with sharded tables.

Counterintuitive Finding 4: Benchmarks Uncovered a Real Bug in Permission Validation

Three answers came back blank. Investigation revealed that the SQL scope check misread periods in WITH RECURSIVE periods(n) AS (...) as a table the caller had no permission to access, and thus refused to execute. It failed closed (no data leakage), but every recursive query from every user was incorrectly rejected. The issue has been fixed and tests added.

The author's assessment: benchmarks you didn't write yourself are exactly what excels at finding this kind of problem. The lesson generalizes beyond SQL—while "fail closed" permission validation is safe, silent rejection masks functional defects and requires dedicated test cases to expose.

All things considered, schemagate is open-sourced under Apache-2.0, installable via pip install schemagate, and the benchmark page lists all caveats. For teams building SQL Agents, the real value of these results is not the scores themselves, but the reusable evaluation discipline they demonstrate: split the data before tuning, use external benchmarks to check your own logic, and remain skeptical of the default advantages of external models.

One-line Summary

The schemagate author benchmarked the SQL Agent table-selection layer on BIRD and Spider 2.0, uncovering four counterintuitive conclusions.

Paragraph Summary

The author of the open-source project schemagate published benchmark results for its SQL Agent schema selection layer on BIRD and Spider 2.0: 77.1% execution accuracy on the BIRD dev set, 71.9% on the Spider 2.0-lite SQLite subset, and only 45.9% on the BigQuery subset. The tests revealed four counterintuitive findings: the larger Opus actually loses to Sonnet on Spider 2.0, showing model selection must be based on testing on your own data; multi-candidate voting plus a judge showed about +3 points on the tuning set but zero gain on the holdout set at 5.5× the cost; BigQuery's 366 date-sharded tables fill the selection slots and cause the model to refuse to answer, which collapsing into wildcard tables solves; and external benchmarks also exposed a real bug where WITH RECURSIVE was misjudged as unauthorized access. The author emphasizes the evaluation discipline of splitting data before tuning and using external benchmarks to check your own logic.