MBPP
MBPP (Mostly Basic Python Problems) is a Python programming benchmark for evaluating code generation capabilities, released by Google Research. The benchmark contains hundreds of Python programming problems ranging from introductory to intermediate difficulty, each accompanied by a natural language description, a reference solution, and test cases. It is used to measure large language models' ability to interpret programming requirements and generate correct Python code, and is a widely used standard benchmark in the code generation domain.
Core Facts
MBPP (Mostly Basic Python Problems) was released by Google and contains approximately 1,000 entry-level programming problems.
Mainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
Code large language models are evaluated using executable benchmarks such as HumanEval and MBPP, meaning the generated code is directly executed and the output is checked for correctness.
Timeline (last 90 days)
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
On MBPP, only 10 problems matched removed data, while the two original-data runs differed by 11 problems; the published best of 38.9% is best understood as a lucky random seed, and the run-to-run variance on MBPP is about 4 points.
FrontierCode is closer to the complexity of production environments compared to single-function completion benchmarks such as HumanEval or MBPP.
CursorBench is a vertical-scenario evaluation benchmark, distinct from general-purpose code generation benchmarks such as HumanEval and MBPP.
Vertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
The pass@k metric is widely adopted on standard benchmarks such as HumanEval and MBPP.
HumanEval and MBPP primarily evaluate single-function-level code generation and have relatively limited discriminating power for agent-type tools.
Pass rates on HumanEval and MBPP are generally far higher than on feature-level benchmarks, but they have been criticized for failing to reflect the real challenges of production environments.
HumanEval and MBPP are earlier, finer-grained code generation benchmarks that primarily assess single-function implementation.
The Qwen model has continuously set new records on programming benchmarks such as HumanEval and MBPP, as well as mathematical reasoning tests such as GSM8K, becoming a performance benchmark among open-source models.
6 more timeline events
All Facts (20)
MBPP (Mostly Basic Python Problems) was released by Google and contains approximately 1,000 entry-level programming problems.
80%VerifiedMainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedCode large language models are evaluated using executable benchmarks such as HumanEval and MBPP, meaning the generated code is directly executed and the output is checked for correctness.
75%VerifiedThe most recognized evaluation benchmarks in the AI programming field include HumanEval (testing the generation of correct implementations from function docstrings) and SWE-bench (locating and fixing actual issues in real GitHub repositories).
65%VerifiedThe training data sources for specialized programming models include open-source code from GitHub, competitive programming solutions, technical documentation, and code review records, and the models are optimized for benchmarks such as HumanEval, MBPP, and SWE-bench.
65%UnverifiedCodex模型的评估通常使用HumanEval和MBPP等编码基准测试
85%UnverifiedHumanEval and MBPP primarily measure a model's code generation capability, while code review is an essentially different task, and the industry currently lacks standardized code review benchmarks.
60%UnverifiedOn MBPP, only 10 problems matched removed data, while the two original-data runs differed by 11 problems; the published best of 38.9% is best understood as a lucky random seed, and the run-to-run variance on MBPP is about 4 points.
50%UnverifiedAfter decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
50%UnverifiedFrontierCode is closer to the complexity of production environments compared to single-function completion benchmarks such as HumanEval or MBPP.
50%UnverifiedCursorBench is a vertical-scenario evaluation benchmark, distinct from general-purpose code generation benchmarks such as HumanEval and MBPP.
50%UnverifiedVertical benchmarks include HumanEval/MBPP (code), MATH/GSM8K (mathematical reasoning), and MMLU (multidisciplinary knowledge).
50%UnverifiedThe pass@k metric is widely adopted on standard benchmarks such as HumanEval and MBPP.
50%UnverifiedHumanEval and MBPP primarily evaluate single-function-level code generation and have relatively limited discriminating power for agent-type tools.
50%UnverifiedHumanEval and MBPP are earlier, finer-grained code generation benchmarks that primarily assess single-function implementation.
50%UnverifiedPass rates on HumanEval and MBPP are generally far higher than on feature-level benchmarks, but they have been criticized for failing to reflect the real challenges of production environments.
50%UnverifiedBenchmarks widely used in the current AI field include HumanEval and MBPP for coding ability, GSM8K and MATH for mathematical reasoning, and MMLU for overall capability.
50%UnverifiedMBPP contains 974 entry-level Python programming problems.
50%UnverifiedExisting model evaluation frameworks (such as HumanEval, MBPP, and various Arena benchmarks) primarily measure capability ceilings and rarely incorporate end-to-end task completion rate as an evaluation dimension.
50%Unverified智谱GLM-4系列在HumanEval、MBPP等代码基准测试中达到国际第一梯队水平
50%