MMLU-Pro
An enhanced version of the original MMLU benchmark that expands the number of answer choices from four to ten and incorporates more questions requiring deep reasoning rather than recall. These modifications make it substantially harder to achieve high scores through guessing alone.
Timeline (last 90 days)
在全部 38 项基准中，Aplomb 1 在 8 项上拿到 5.3B 以内模型最高分，包括 MMLU-Pro、GPQA Diamond 和 BBH
Version 2.0 shows declines in knowledge-based capabilities (MMLU-Pro 72.5 vs. base 78.5) and competition-level coding ability (LiveCodeBench 51 vs. 56).
The evaluation covered five benchmarks: MMLU-Pro, GPQA Diamond, GSM8K, German MGSM, and IFEval.
On the MMLU-Pro benchmark, Flash-Next IQ4_XS scored 83.8% and Qwen3 27B FP8 scored 75.0%.
The score gap among multiple frontier models on MMLU has narrowed to the order of one percentage point, driving the emergence of more challenging benchmarks such as MMLU-Pro and ARC-AGI.
All Facts (4)
在全部 38 项基准中，Aplomb 1 在 8 项上拿到 5.3B 以内模型最高分，包括 MMLU-Pro、GPQA Diamond 和 BBH
50%UnverifiedVersion 2.0 shows declines in knowledge-based capabilities (MMLU-Pro 72.5 vs. base 78.5) and competition-level coding ability (LiveCodeBench 51 vs. 56).
50%UnverifiedThe evaluation covered five benchmarks: MMLU-Pro, GPQA Diamond, GSM8K, German MGSM, and IFEval.
50%UnverifiedThe score gap among multiple frontier models on MMLU has narrowed to the order of one percentage point, driving the emergence of more challenging benchmarks such as MMLU-Pro and ARC-AGI.
50%