HumanEval
HumanEval is a code generation benchmark released by OpenAI, used to evaluate a model's ability to produce functionally correct code from function signatures and docstrings.
Core Facts
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
HumanEval was released by OpenAI and contains 164 hand-crafted Python programming problems, with pass@k as its core metric.
HumanEval was proposed by OpenAI in 2021 and evaluates function-level code completion.
Timeline (last 90 days)
In the Magicoder-OSS-Instruct-75K dataset, among 75,197 samples, 0 HumanEval problems were found with docstring matches, and 1 problem had a reference answer match (1 case).
A 360M-parameter code model trained by the trifero.ai lab under Strife Technologies initially reported 26.8% pass@1 on HumanEval, and after removing instruction data overlapping with the benchmark, the true number was 22.6%.
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
The decontaminated model was about 4 points lower than the mean of the two original-data runs on both the "in removed data" and "not in removed data" HumanEval subsets, indicating that the contamination mechanism is not memorization but rather a general boost to the model's performance on HumanEval-style problems.
Contamination detection is performed on each instruction set using two checks: the problem text is normalized by taking the first 80 characters of the HumanEval docstring and compared; the reference answer, after removing comments and docstrings, is checked for whether more than half of the 13-token n-grams match.
Even in the Magicoder-Evol-Instruct-110K dataset labeled as "decontaminated," 52 of the 164 HumanEval problems still had docstring matches, and 8 had answer matches.
Code generation has mature evaluation sets like HumanEval and SWE-bench, but the code review scenario is different, and existing general benchmarks struggle to cover the dimensions required for review.
A well-calibrated model is one whose expressed confidence matches its actual accuracy, whereas mainstream leaderboards such as MMLU and HumanEval typically measure only accuracy without penalizing overconfidence.
HumanEval tests whether a model can complete isolated function snippets, whereas real-world engineering more commonly involves locating issues in large codebases, understanding business logic, and making cross-file modifications.
Mainstream benchmarks such as MMLU, HumanEval, and GSM8K were originally designed primarily for academic comparison, focusing on measuring model accuracy on closed-form questions.
40 more timeline events
All Facts (20)
MMLU evaluates a model's breadth of knowledge across 57 subject areas, HumanEval measures code generation accuracy, GSM8K tests mathematical reasoning, and HELM constructs a comprehensive multi-dimensional evaluation framework.
90%VerifiedHumanEval was released by OpenAI and contains 164 hand-crafted Python programming problems, with pass@k as its core metric.
90%VerifiedHumanEval was proposed by OpenAI in 2021 and evaluates function-level code completion.
90%VerifiedHumanEval was proposed by OpenAI and contains 164 Python programming problems, primarily testing function-level code generation capability
90%VerifiedThe DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
85%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedHumanEval was developed in-house by OpenAI and is specifically designed to test the functional correctness rate of code generation.
85%VerifiedStandard AI benchmarks include MMLU (a comprehensive knowledge test covering 57 subjects), GPQA (graduate-level professional question answering), SWE-bench (real-world software engineering tasks), HumanEval and MBPP (programming ability), and MATH and GSM8K (mathematical reasoning)
85%VerifiedMMLU contains approximately 15,000 multiple-choice questions across 57 subject areas; HumanEval, designed by OpenAI, contains 164 Python programming problems; MATH contains 12,500 competition math problems; and GPQA covers graduate-level science questions.
80%VerifiedCredible capability evaluation of models in the AI field relies on standardized benchmark tests; commonly used evaluation sets include MMLU, HumanEval, MATH, and GPQA.
80%VerifiedPass rates of code generation models on benchmarks such as HumanEval jumped from 60% to over 90% between 2024 and 2025
80%VerifiedHumanEval是由OpenAI于2021年发布的代码生成基准测试，包含164个手写的Python编程问题
80%Verified大语言模型通过在数千亿乃至数万亿个词元（Token）的文本数据上进行自监督预训练
80%VerifiedMainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedClaude 3.5 Sonnet is a model released by Anthropic in 2024 that surpassed the contemporaneous GPT-4o on the code generation benchmark HumanEval
80%VerifiedOn mainstream evaluations such as MMLU, HumanEval, and GSM8K, the score gaps among leading large models are often only a few percentage points.
80%VerifiedBy mid-2024, top models had generally exceeded scores of 85–90% on MMLU, causing the benchmark's discriminative power to drop sharply.
75%VerifiedSome models have already surpassed 90% of the human average score on MMLU, causing their discriminative power to drop sharply.
75%VerifiedGPT-4 and Claude 3 achieve pass@1 rates exceeding 85% on the HumanEval code benchmark.
75%VerifiedModel evaluations in the code generation field, such as HumanEval and SWE-bench, place special emphasis on functional correctness rather than fluency.
75%