GPQA
GPQA (Graduate-Level Google-Proof Q&A) is a professional knowledge question-answering benchmark at the graduate level, designed to evaluate large language models' deep reasoning and domain knowledge in natural sciences including biology, chemistry, and physics. Questions are crafted by domain experts, are of high difficulty, and are not easily answered even with the aid of search engines. The benchmark is commonly used to measure models' reasoning performance at the limits of specialized domains.
Core Facts
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
Credible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
DeepSeek V4.1 Flash scores lower than the previous version on GPQA, HLE, and other benchmarks.
Timeline (last 90 days)
Commonly used evaluation benchmarks for measuring the gap between open-source and closed-source models include HumanEval/SWE-bench for coding ability, MATH and GPQA for reasoning ability, and MMLU for overall performance.
Astra scored lower than small models such as GLM 5.3 Flash on the Artificial Analysis intelligence index, because that index is based on outdated and saturated benchmarks such as GPQA.
HumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
Commonly used benchmarks for evaluating large language model capabilities include MMLU, HumanEval, MATH, and GPQA.
LiveNerf reruns a set of independent benchmarks every day, such as GPQA and SWE-bench, and plots the results in charts recorded in a GitHub repository.
Third-party platforms such as Artificial Analysis normalize scores from multiple sub-benchmarks and aggregate them into a single quality index, enabling different models to be compared on the same scale.
Common sub-benchmarks included in composite intelligence indices are MMLU, HumanEval, MBPP, MATH, GSM8K, GPQA, and LongBench.
MMLU, GPQA, and SWE-bench are model evaluation benchmarks widely recognized by the current AI community.
GPQA (Graduate-Level Google-Proof Q&A) focuses on doctoral-level scientific reasoning problems.
To evaluate whether a local model is suitable for your needs, you can refer to standardized benchmark scores such as MMLU and HumanEval.
14 more timeline events
All Facts (20)
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedCredible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
80%VerifiedDeepSeek V4.1 Flash scores lower than the previous version on GPQA, HLE, and other benchmarks.
75%VerifiedGPQA is a benchmark designed for PhD-level scientific question answering, with questions difficult enough that domain experts can only answer approximately 65% correctly.
70%VerifiedGPQA (Graduate-Level Google-Proof Q&A) focuses on questions requiring doctoral-level domain expertise.
65%UnverifiedThe industry has begun shifting toward more challenging benchmarks like GPQA and MATH as older benchmarks lose discriminative power
80%UnverifiedGPQA (Graduate-Level Google-Proof Q&A) focuses on doctoral-level scientific reasoning problems.
60%UnverifiedGPQA is a graduate-level scientific question-answering benchmark, and SWE-bench targets real-world software engineering tasks.
60%UnverifiedGPQA was designed by a research team at New York University; non-experts using search engines achieve an accuracy of approximately 34%, while domain experts average approximately 65%.
60%UnverifiedCommonly used evaluation benchmarks for measuring the gap between open-source and closed-source models include HumanEval/SWE-bench for coding ability, MATH and GPQA for reasoning ability, and MMLU for overall performance.
50%UnverifiedAstra scored lower than small models such as GLM 5.3 Flash on the Artificial Analysis intelligence index, because that index is based on outdated and saturated benchmarks such as GPQA.
50%UnverifiedHumanEval and SWE-bench are commonly used to evaluate large model coding capabilities, MMLU and GPQA for comprehensive reasoning, and IFEval for instruction following.
50%UnverifiedCommonly used benchmarks for evaluating large language model capabilities include MMLU, HumanEval, MATH, and GPQA.
50%UnverifiedLiveNerf reruns a set of independent benchmarks every day, such as GPQA and SWE-bench, and plots the results in charts recorded in a GitHub repository.
50%UnverifiedCommon sub-benchmarks included in composite intelligence indices are MMLU, HumanEval, MBPP, MATH, GSM8K, GPQA, and LongBench.
50%UnverifiedThird-party platforms such as Artificial Analysis normalize scores from multiple sub-benchmarks and aggregate them into a single quality index, enabling different models to be compared on the same scale.
50%UnverifiedMMLU, GPQA, and SWE-bench are model evaluation benchmarks widely recognized by the current AI community.
50%UnverifiedTo evaluate whether a local model is suitable for your needs, you can refer to standardized benchmark scores such as MMLU and HumanEval.
50%UnverifiedHigh-difficulty benchmarks that have emerged in recent years include GPQA (graduate-level science question answering), SWE-bench (real software engineering tasks), and various agentic evaluations.
50%UnverifiedDeepSeek V4.1 Flash falls short on purely academic and purely knowledge-based tasks (such as GPQA) and is not suitable for deep academic research or complex writing tasks.
50%