Spider
Spider is a large-scale cross-domain benchmark dataset for Text-to-SQL tasks, designed to evaluate the ability of natural language processing models to convert natural language questions into structured SQL queries. The dataset covers databases from multiple domains, includes complex multi-table join scenarios, and requires cross-domain generalization, serving as a widely referenced standard for measuring Text-to-SQL system performance in both academia and industry.
Core Facts
The Spider benchmark was released by Yale University in 2018 and contains more than 10,000 natural language questions and corresponding SQL queries, covering 200 databases.
In mainstream Text-to-SQL benchmarks such as Spider and BIRD, top models have already exceeded 85% accuracy.
Timeline (last 90 days)
The commonly used benchmarks for measuring NL2SQL system accuracy are the Spider and BIRD datasets, where Spider tests cross-domain generalization ability and BIRD more closely reflects the complexity of real enterprise databases.
On the Spider benchmark, the accuracy of state-of-the-art models rose from under 20% in 2018 to over 80%
In mainstream Text-to-SQL benchmarks such as Spider and BIRD, top models have already exceeded 85% accuracy.
All Facts (9)
The Spider benchmark was released by Yale University in 2018 and contains more than 10,000 natural language questions and corresponding SQL queries, covering 200 databases.
85%VerifiedIn mainstream Text-to-SQL benchmarks such as Spider and BIRD, top models have already exceeded 85% accuracy.
75%UnverifiedOn the Spider benchmark, the accuracy of state-of-the-art models rose from under 20% in 2018 to over 80%
85%UnverifiedIRNet和RATSQL等基于Seq2Seq和Transformer的模型在Spider基准数据集上取得了Text-to-SQL的突破
85%UnverifiedGPT-4等大模型在Spider等基准测试上的零样本或少样本SQL生成准确率已超过80%
75%Unverified在Spider基准上，早期单模型方案准确率约为70%，GPT-4等大模型将其提升到80%以上，多智能体协作策略的系统已能达到85%甚至更高的执行准确率
75%UnverifiedMainstream Text-to-SQL solutions rely on models fine-tuned on standard benchmark datasets such as WikiSQL and Spider.
70%UnverifiedThe commonly used benchmarks for measuring NL2SQL system accuracy are the Spider and BIRD datasets, where Spider tests cross-domain generalization ability and BIRD more closely reflects the complexity of real enterprise databases.
50%UnverifiedSpider爬虫工具用Rust原生编写，处理速度比同类产品快7倍
50%