BIRD
A tool or method for tracing and analyzing the reasoning processes of large language models, designed to identify and localize the causes of failures that occur during reasoning execution.
Timeline (last 90 days)
In the BIRD benchmark, Aura's token-substitution approach achieved an accuracy of 67.6%, only 0.6 percentage points below the 68.2% achieved with real data, while the number of leaked personal information items dropped from 849 to 0.
The BIRD benchmark covers 179 questions across 4 databases, with queries generated by Google's gemini-3.8-flash.
The approach of using a purely local small model that does not send data externally achieved only 35.2% accuracy on the BIRD test, far below the tokenized cloud-based approach.
BIRD is a text-to-SQL benchmark published by the University of Hong Kong in collaboration with Google and other institutions, incorporating real-world challenges such as dirty data, numerical reasoning, and external knowledge dependencies.
The commonly used benchmarks for measuring NL2SQL system accuracy are the Spider and BIRD datasets, where Spider tests cross-domain generalization ability and BIRD more closely reflects the complexity of real enterprise databases.
The study uses BIRD tracing to localize failures to three stages: mechanism implementation, activation, and output validity.
BIRD-financial is the financial subset of the BIRD benchmark, with SQL answers written by human experts, and is widely used to evaluate text-to-SQL model performance
All Facts (7)
In the BIRD benchmark, Aura's token-substitution approach achieved an accuracy of 67.6%, only 0.6 percentage points below the 68.2% achieved with real data, while the number of leaked personal information items dropped from 849 to 0.
50%UnverifiedThe BIRD benchmark covers 179 questions across 4 databases, with queries generated by Google's gemini-3.8-flash.
50%UnverifiedThe approach of using a purely local small model that does not send data externally achieved only 35.2% accuracy on the BIRD test, far below the tokenized cloud-based approach.
50%UnverifiedBIRD is a text-to-SQL benchmark published by the University of Hong Kong in collaboration with Google and other institutions, incorporating real-world challenges such as dirty data, numerical reasoning, and external knowledge dependencies.
50%UnverifiedThe commonly used benchmarks for measuring NL2SQL system accuracy are the Spider and BIRD datasets, where Spider tests cross-domain generalization ability and BIRD more closely reflects the complexity of real enterprise databases.
50%UnverifiedThe study uses BIRD tracing to localize failures to three stages: mechanism implementation, activation, and output validity.
50%UnverifiedBIRD-financial is the financial subset of the BIRD benchmark, with SQL answers written by human experts, and is widely used to evaluate text-to-SQL model performance
50%