Massive Multitask Language Understanding，大规模多任务语言理解基准测试，用于评估大语言模型的知识和推理能力
众包式盲测评测被认为比传统学术基准测试（如MMLU、HumanEval）更能反映模型在开放式对话中的综合表现
冻结基准的设计针对AI评测领域的数据污染问题，MMLU、HumanEval等知名基准都面临数据污染困境
MMLU contains approximately 15,000 multiple-choice questions across 57 subjects, covering knowledge from high school to graduate level
MMLU was introduced in 2021 by UC Berkeley and other institutions as a large-scale multitask language understanding benchmark
Models like GPT-4 and Claude 3.5 surpassed 90% accuracy on MMLU, significantly diminishing its discriminative power
MMLU包含57个学科领域约1.5万道选择题，HumanEval由OpenAI设计包含164道Python编程题，MATH包含12,500道竞赛数学题，GPQA收录研究生级别科学问题
当前AI模型评测主要依赖标准化基准测试（如MMLU、HumanEval、GSM8K等），但存在数据污染风险和与实际应用脱节的问题
在MMLU、HumanEval、GSM8K等主流评测中，头部大模型的得分差距往往只有几个百分点
MMLU contains approximately 15,000 multiple-choice questions across 57 subjects, covering knowledge from high school to graduate level
90%待验证MMLU was introduced in 2021 by UC Berkeley and other institutions as a large-scale multitask language understanding benchmark
90%待验证Models like GPT-4 and Claude 3.5 surpassed 90% accuracy on MMLU, significantly diminishing its discriminative power
85%待验证当前AI模型评测主要依赖标准化基准测试（如MMLU、HumanEval、GSM8K等），但存在数据污染风险和与实际应用脱节的问题
80%待验证在MMLU、HumanEval、GSM8K等主流评测中，头部大模型的得分差距往往只有几个百分点
70%待验证MMLU包含57个学科领域约1.5万道选择题，HumanEval由OpenAI设计包含164道Python编程题，MATH包含12,500道竞赛数学题，GPQA收录研究生级别科学问题
50%待验证冻结基准的设计针对AI评测领域的数据污染问题，MMLU、HumanEval等知名基准都面临数据污染困境
50%待验证众包式盲测评测被认为比传统学术基准测试（如MMLU、HumanEval）更能反映模型在开放式对话中的综合表现
50%