GSM8K
GSM8K is a benchmark comprising approximately 8,500 grade-school-level math word problems, used to evaluate the basic mathematical reasoning capabilities of large language models.
Core Facts
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
Standard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
Timeline (last 90 days)
Traditional benchmarks such as MMLU and GSM8K focus on language and mathematics, while physical intuition tests require models to predict the state of objects after being subjected to forces, collisions, and stacking.
These flaws of large language models are often masked by high scores in standardized benchmarks (such as MMLU, GSM8K), because the evaluation sets have fixed questions and standardized formats and do not involve the ambiguity and noise of real-world scenarios.
Getting the continuous RL pipeline working on GSM8K only proves the mechanism is feasible, not that it applies to real business tasks with ambiguous reward signals, because most real tasks have no ready-made programmatic scorer.
reef's OpenClaw-RL recipe uses 72 GSM8K problems as the task stream, with Qwen3-4B serving simultaneously as the policy model and the process reward model, Qwen3-32B as the student model, using seven GPUs, and setting a threshold that must be reached before an update is accepted.
Traditional benchmarks such as MMLU and GSM8K focus on academic ability, with fixed questions that are easily covered by training data, leading to data contamination problems.
On the GSM8K mathematical reasoning test, Qwen3-4B scores 0.915 and Gemma 4 E2B scores 0.911, a difference of +0.4 percentage points with a confidence interval crossing zero, making it the only model on par with Gemma.
The core flaw of traditional static benchmarks (such as MMLU and GSM8K) is their one-shot nature—once the model outputs an answer the task is over, with no opponent counterplay, information updates, or strategy adjustment, measuring single-step accuracy rather than the quality of continuous decision-making.
Even in large-scale pretraining corpora that have undergone deduplication, n-gram overlap rates with mainstream benchmarks such as MMLU, HellaSwag, and GSM8K can still reach several percentage points.
Mainstream benchmarks such as MMLU, HumanEval, and GSM8K were originally designed primarily for academic comparison, focusing on measuring model accuracy on closed-form questions.
PAL has the model generate a Python program that is then executed by an interpreter to produce the answer, achieving an accuracy rate approximately 10–15 percentage points higher than pure text Chain-of-Thought on the GSM8K benchmark
40 more timeline events
All Facts (20)
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
90%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedMMLU contains approximately 15,000 multiple-choice questions across 57 subject areas; HumanEval, designed by OpenAI, contains 164 Python programming problems; MATH contains 12,500 competition math problems; and GPQA covers graduate-level science questions.
80%VerifiedGSM8K is an elementary school math word problem benchmark released by OpenAI in 2021, containing approximately 8,500 problems that require 2 to 8 reasoning steps
80%VerifiedOn mainstream evaluations such as MMLU, HumanEval, and GSM8K, the score gaps among leading large models are often only a few percentage points.
80%VerifiedGSM8K, released by OpenAI, contains approximately 8,500 elementary school math word problems designed to test multi-step reasoning ability
75%VerifiedHumanEval was designed by OpenAI and contains 164 Python programming problems; GSM8K contains 8,500 elementary school math word problems; MMLU covers multiple-choice questions across 57 subject areas.
75%VerifiedAI model benchmarking typically evaluates a model's reasoning, coding, and mathematical capabilities on standard datasets such as MMLU, HumanEval, and GSM8K.
70%VerifiedWhen test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.
65%VerifiedCommonly used AI benchmarks in the industry include MMLU (Massive Multitask Language Understanding), HumanEval (code generation), and GSM8K (mathematical reasoning), but each benchmark can only measure capabilities along specific dimensions.
65%VerifiedMMLU, HumanEval, and MT-Bench are standardized benchmark test sets widely recognized by the AI community.
65%UnverifiedOn mainstream benchmarks such as MMLU, HumanEval, and GSM8K, open-source models have reached 85–95% of the level of early versions of GPT-4.
70%UnverifiedThe Self-Consistency method, proposed by Xuezhi Wang and others, improved accuracy by more than 10% on mathematical reasoning benchmarks such as GSM8K.
70%Partially VerifiedGSM8K是小学数学推理基准，HumanEval是代码生成基准，HellaSwag是常识推理基准
65%UnverifiedMMLU covers 57 subjects and nearly 16,000 multiple-choice questions; HumanEval was developed by OpenAI and contains 164 programming problems; GSM8K contains 8,500 elementary school math word problems.
60%UnverifiedHumanEval consists of 164 Python programming problems, and GSM8K contains 8,500 elementary school math word problems.
60%UnverifiedCurrent AI model evaluation relies primarily on standardized benchmark tests (such as MMLU, HumanEval, and GSM8K), but these face risks of data contamination and disconnection from real-world applications.
60%UnverifiedOn the GSM8K mathematical reasoning benchmark, Self-Consistency can improve accuracy by approximately 17 percentage points compared to single greedy decoding.
60%UnverifiedTraditional benchmarks such as MMLU and GSM8K focus on language and mathematics, while physical intuition tests require models to predict the state of objects after being subjected to forces, collisions, and stacking.
50%