BIG-Bench
A large-scale language model evaluation benchmark released by Google in 2022. It systematically documents various emergent phenomena across hundreds of tasks and is used to measure the capabilities of large language models on diverse task types.
Timeline (last 90 days)
Benchmark saturation phenomenon: after ImageNet was conquered, ObjectNet and ImageNet-A emerged; after GLUE was saturated, SuperGLUE appeared, followed by BIG-Bench and BIG-Bench Hard.
Benchmarks being led and established by the model's publishing party is a common phenomenon in the current large model field, with similar situations occurring in OpenAI's SimpleQA, Google's BIG-Bench, and other evaluations.
The HumanEval series focuses on tracking code capabilities, while BIG-Bench is an academic evaluation project.
Academia typically relies on standardized evaluation sets maintained by independent organizations, such as GLUE, SuperGLUE, and BIG-Bench, to compare model performance across the board.
The BIG-Bench evaluation system released by Google in 2022 documented various emergent phenomena and sparked widespread discussion.
Mainstream large language model evaluation systems such as MMLU, HellaSwag, and BIG-Bench primarily measure knowledge coverage, logical reasoning, and linguistic fluency, with the design assumption that there are verifiable correct answers.
All Facts (6)
Benchmark saturation phenomenon: after ImageNet was conquered, ObjectNet and ImageNet-A emerged; after GLUE was saturated, SuperGLUE appeared, followed by BIG-Bench and BIG-Bench Hard.
50%UnverifiedBenchmarks being led and established by the model's publishing party is a common phenomenon in the current large model field, with similar situations occurring in OpenAI's SimpleQA, Google's BIG-Bench, and other evaluations.
50%UnverifiedThe HumanEval series focuses on tracking code capabilities, while BIG-Bench is an academic evaluation project.
50%UnverifiedAcademia typically relies on standardized evaluation sets maintained by independent organizations, such as GLUE, SuperGLUE, and BIG-Bench, to compare model performance across the board.
50%UnverifiedThe BIG-Bench evaluation system released by Google in 2022 documented various emergent phenomena and sparked widespread discussion.
50%UnverifiedMainstream large language model evaluation systems such as MMLU, HellaSwag, and BIG-Bench primarily measure knowledge coverage, logical reasoning, and linguistic fluency, with the design assumption that there are verifiable correct answers.
50%