MATH
MATH is a benchmark dataset for evaluating mathematical reasoning in machine learning models, constructed by Hendrycks et al. It contains thousands of problems drawn from high school mathematics competitions, spanning algebra, geometry, number theory, probability, and other domains, with problems stratified by difficulty level. Each problem requires the model to produce a complete solution process along with a final answer, making it a standard benchmark for assessing complex mathematical reasoning in large language models.
Core Facts
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
HumanEval focuses on code generation capability, testing models' logical reasoning and code correctness with 164 programming problems.
Some models carry the risk of over-optimization toward test sets, causing leaderboard scores to deviate from real-world performance (benchmark saturation).
Timeline (last 90 days)
Artificial Analysis's Intelligence Index is a composite score derived from a weighted average of a series of existing benchmarks (such as MMLU, HumanEval, MATH, etc.).
The cost-performance curve, which typically uses benchmark scores such as MMLU, MATH, and HumanEval on the vertical axis and cost per million tokens on the horizontal axis, is a visualization tool commonly used by third-party evaluation platforms such as Artificial Analysis and LMSys.
Benchmarks commonly used by the open-source community (such as MMLU, HumanEval, MATH, and LongBench) carry risks of data contamination.
DeepSeek-R1 approaches or matches GPT-4-level performance on standardized benchmarks such as MATH and HumanEval.
Mainstream large model benchmarks such as MMLU, BigBench, and MATH have all been widely used in training data filtering and reinforcement learning reward signal design, making it increasingly difficult for high scores to map to real capability improvements.
Commonly used benchmarks for evaluating large language model capabilities include MMLU, HumanEval, MATH, and GPQA.
Common public benchmarks such as MMLU focus on multi-subject question answering, HumanEval on code generation, and MATH on mathematical reasoning, each emphasizing different dimensions.
Vertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
The GPT-4o, o1, and o3 series have continuously set records on specific benchmarks (such as MATH, HumanEval, and MMLU), yet many developers find it difficult to perceive improvements of the same magnitude in everyday use.
DeepSeek-Math 7B is an open-source mathematics large model with 7B parameters, whose RL version with tool invocation can achieve approximately 60% accuracy on the MATH benchmark, approaching GPT-4 level.
16 more timeline events
All Facts (20)
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedHumanEval focuses on code generation capability, testing models' logical reasoning and code correctness with 164 programming problems.
85%VerifiedSome models carry the risk of over-optimization toward test sets, causing leaderboard scores to deviate from real-world performance (benchmark saturation).
70%VerifiedCommonly used AI benchmarks in the industry include MMLU (Massive Multitask Language Understanding), HumanEval (code generation), and GSM8K (mathematical reasoning), but each benchmark can only measure capabilities along specific dimensions.
65%VerifiedArtificial Analysis Intelligence Index整合了MMLU、HumanEval、MATH、GPQA等多个基准测试的加权结果
65%VerifiedBenchmarks such as MMLU, HumanEval, and MATH are used to quantify differences in model capabilities.
65%VerifiedMMLU, HumanEval, and MT-Bench are standardized benchmark test sets widely recognized by the AI community.
65%UnverifiedThe industry has begun shifting toward more challenging benchmarks like GPQA and MATH as older benchmarks lose discriminative power
80%UnverifiedOpenAI's 2023 paper "Let's Verify Step by Step" demonstrated that process supervision (PRM) significantly outperforms outcome supervision (ORM) on mathematical reasoning tasks.
70%Partially VerifiedGSM8K是小学数学推理基准，HumanEval是代码生成基准，HellaSwag是常识推理基准
65%UnverifiedThere is often a significant gap between benchmark scores and real-world application performance; a model with a slightly lower MMLU score may perform better in specific vertical domains.
60%UnverifiedArtificial Analysis's Intelligence Index is a composite score derived from a weighted average of a series of existing benchmarks (such as MMLU, HumanEval, MATH, etc.).
50%UnverifiedThe cost-performance curve, which typically uses benchmark scores such as MMLU, MATH, and HumanEval on the vertical axis and cost per million tokens on the horizontal axis, is a visualization tool commonly used by third-party evaluation platforms such as Artificial Analysis and LMSys.
50%UnverifiedBenchmarks commonly used by the open-source community (such as MMLU, HumanEval, MATH, and LongBench) carry risks of data contamination.
50%UnverifiedDeepSeek-R1 approaches or matches GPT-4-level performance on standardized benchmarks such as MATH and HumanEval.
50%UnverifiedMainstream large model benchmarks such as MMLU, BigBench, and MATH have all been widely used in training data filtering and reinforcement learning reward signal design, making it increasingly difficult for high scores to map to real capability improvements.
50%UnverifiedCommonly used benchmarks for evaluating large language model capabilities include MMLU, HumanEval, MATH, and GPQA.
50%UnverifiedCommon public benchmarks such as MMLU focus on multi-subject question answering, HumanEval on code generation, and MATH on mathematical reasoning, each emphasizing different dimensions.
50%UnverifiedVertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
50%UnverifiedThe GPT-4o, o1, and o3 series have continuously set records on specific benchmarks (such as MATH, HumanEval, and MMLU), yet many developers find it difficult to perceive improvements of the same magnitude in everyday use.
50%