BIRD

Both 92.9% and 8.8% are medians from the author's own testing, and were obtained only on specific schemas in Spider 2.0 and BIRD; no third-party independent reproduction has yet been seen.

On BIRD's seven-table database, SchemagateRetriever reduced prompt size by only 8.8%.

BIRD is one of the most influential public benchmarks in the Text-to-SQL field, its dev set contains 1534 natural-language questions across multiple databases, and evaluation is typically based on execution accuracy.

On the same pipeline and the same 135 Spider 2.0-lite SQLite questions, Sonnet got 71.9% while Opus got only 67.4%, but Opus performed better on BIRD.

On all 1534 questions of the BIRD dev set, Claude Opus 5.5 achieved 77.1% execution accuracy.

UniQL is a new benchmark that translated 1,534 questions from the BIRD dataset into 16 SQL dialects.

By adding to the prompt 'you are seeing part of this database; if you need more information, refuse to answer', schemagate reduced incorrect answers from 16% to 5% in a 100-question pilot.

The schemagate authors note its limitations: only one model was tested, and the prompt wording was chosen on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08).

The paper tested the best pure schema-filtering configuration on the BIRD dataset, and still about 27% of questions that should have been refused were answered by the model.

Providing the model with BIRD's column descriptions actually made schemagate's results worse.

On the BIRD dataset, the paper's best "hidden schema" configuration still answered 27% of questions that should have been refused.

The wording of the prompt was chosen on 100 BIRD questions that are part of the test set, so there is tuning contamination.

The BIRD database version is the 2024 edition, while the paper used the 2025 edition.

schemagate's benchmark conditions: all 876 tables of Spider dev and BIRD dev are placed in the same pool, the model does not know which database a question belongs to, and a table must all appear in the set of tables used by the correct SQL to count as a top-5 hit.

For the schemagate benchmark's paired McNemar test, the p-value is about 1e-49 on Spider and about 1e-36 on BIRD.

In schemagate's benchmark, on the Spider dataset's 1034 questions the pure mathematical ranking accuracy is 80.3%, and 96.5% after re-ranking; on the BIRD dataset's 1534 questions the pure mathematical ranking is 88.6%, and 96.7% after re-ranking.

In the BIRD benchmark, Aura's token-substitution approach achieved an accuracy of 67.6%, only 0.6 percentage points below the 68.2% achieved with real data, while the number of leaked personal information items dropped from 849 to 0.

BIRD is a text-to-SQL benchmark published by the University of Hong Kong in collaboration with Google and other institutions, incorporating real-world challenges such as dirty data, numerical reasoning, and external knowledge dependencies.

The approach of using a purely local small model that does not send data externally achieved only 35.2% accuracy on the BIRD test, far below the tokenized cloud-based approach.

The BIRD benchmark covers 179 questions across 4 databases, with queries generated by Google's gemini-3.8-flash.