Goodhart's Law
A classic principle in economics and AI: when a measure becomes a target, it ceases to be a good measure of the original goal. In AI alignment, it is commonly used to explain phenomena such as reward hacking and other forms of objective misspecification.
Core Facts
Reward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
基准过拟合是Goodhart定律在AI评估领域的体现，即衡量指标成为优化目标时便失去衡量有效性
Timeline (last 90 days)
Reinforcement learning is prone to reward hacking, where the model learns to game the reward model, which is academically referred to as Goodhart's Law.
Goodhart's Law dictates that once a specialized benchmark becomes an industry standard, model vendors will optimize specifically for it, causing it to lose its representativeness of real business needs.
In their 2018 paper, Manheim and Garrabrant subdivided Goodhart's Law into four types: regressive, extremal, causal, and adversarial
Reward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
In AI research, Goodhart's Law has been subdivided into four variants: regressional effect, extremal effect, causal misalignment, and adversarial gaming.
All Facts (7)
Reward Hacking is a classic problem in the field of reinforcement learning, whose core lies in the inevitable gap between the optimization objective (reward function) and the designer's true intent, first systematically documented and analyzed by research teams including DeepMind.
65%Verified基准过拟合是Goodhart定律在AI评估领域的体现，即衡量指标成为优化目标时便失去衡量有效性
65%UnverifiedReinforcement learning is prone to reward hacking, where the model learns to game the reward model, which is academically referred to as Goodhart's Law.
50%UnverifiedGoodhart's Law dictates that once a specialized benchmark becomes an industry standard, model vendors will optimize specifically for it, causing it to lose its representativeness of real business needs.
50%UnverifiedIn their 2018 paper, Manheim and Garrabrant subdivided Goodhart's Law into four types: regressive, extremal, causal, and adversarial
50%UnverifiedIn AI research, Goodhart's Law has been subdivided into four variants: regressional effect, extremal effect, causal misalignment, and adversarial gaming.
50%Unverified奖励作弊的理论根源是Goodhart定律：当度量指标成为优化目标时，它就不再是好的度量指标
50%