benchmark gaming
A phenomenon in AI development where models are over-optimized for specific evaluation datasets, resulting in inflated benchmark scores that do not reflect real-world production capability. It represents an instance of Goodhart's Law applied to AI evaluation.
Timeline (last 90 days)
Benchmark gaming is widespread, as models may be exposed to test-set-related data during training or fine-tuning, resulting in inflated scores but insufficient generalization ability.
Benchmark gaming refers to the phenomenon where developers optimize targeted for specific benchmarks (such as adding training data stylistically similar to target benchmark questions during the fine-tuning phase) to inflate scores without improving genuine general capabilities
All Facts (2)
Benchmark gaming is widespread, as models may be exposed to test-set-related data during training or fine-tuning, resulting in inflated scores but insufficient generalization ability.
50%UnverifiedBenchmark gaming refers to the phenomenon where developers optimize targeted for specific benchmarks (such as adding training data stylistically similar to target benchmark questions during the fine-tuning phase) to inflate scores without improving genuine general capabilities
50%