MMLU
Massive Multitask Language Understanding，大规模多任务语言理解基准测试，用于评估大语言模型的知识和推理能力
Timeline (last 90 days)
众包式盲测评测被认为比传统学术基准测试（如MMLU、HumanEval）更能反映模型在开放式对话中的综合表现
冻结基准的设计针对AI评测领域的数据污染问题，MMLU、HumanEval等知名基准都面临数据污染困境
MMLU contains approximately 15,000 multiple-choice questions across 57 subjects, covering knowledge from high school to graduate level
MMLU was introduced in 2021 by UC Berkeley and other institutions as a large-scale multitask language understanding benchmark
Models like GPT-4 and Claude 3.5 surpassed 90% accuracy on MMLU, significantly diminishing its discriminative power
MMLU包含57个学科领域约1.5万道选择题，HumanEval由OpenAI设计包含164道Python编程题，MATH包含12,500道竞赛数学题，GPQA收录研究生级别科学问题
当前AI模型评测主要依赖标准化基准测试（如MMLU、HumanEval、GSM8K等），但存在数据污染风险和与实际应用脱节的问题
在MMLU、HumanEval、GSM8K等主流评测中，头部大模型的得分差距往往只有几个百分点
All Facts (8)
MMLU contains approximately 15,000 multiple-choice questions across 57 subjects, covering knowledge from high school to graduate level
90%UnverifiedMMLU was introduced in 2021 by UC Berkeley and other institutions as a large-scale multitask language understanding benchmark
90%UnverifiedModels like GPT-4 and Claude 3.5 surpassed 90% accuracy on MMLU, significantly diminishing its discriminative power
85%Unverified当前AI模型评测主要依赖标准化基准测试（如MMLU、HumanEval、GSM8K等），但存在数据污染风险和与实际应用脱节的问题
80%Unverified在MMLU、HumanEval、GSM8K等主流评测中，头部大模型的得分差距往往只有几个百分点
70%Unverified众包式盲测评测被认为比传统学术基准测试（如MMLU、HumanEval）更能反映模型在开放式对话中的综合表现
50%Unverified冻结基准的设计针对AI评测领域的数据污染问题，MMLU、HumanEval等知名基准都面临数据污染困境
50%UnverifiedMMLU包含57个学科领域约1.5万道选择题，HumanEval由OpenAI设计包含164道Python编程题，MATH包含12,500道竞赛数学题，GPQA收录研究生级别科学问题
50%