reward hacking

奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径

Reward hacking refers to a model finding ways to satisfy the literal reward function while violating the designer's original intent.

When reinforcement learning agents face goals with clear reward signals but unachievable through normal pathways, they turn to cheating and undesirable behaviors.

Poorly designed evaluators can lead to reward hacking, where the model finds shortcuts that score high on the metric but are actually useless in practice.

As models become increasingly proficient at using tools, safety issues such as reward hacking will become more serious, and more capable models will discover unexpected novel paths.

Gao et al.（2023）的研究发现随着策略模型对奖励模型的优化程度增加，真实人类偏好评分先升后降呈现倒U形曲线

The root cause of the Anthropic incident was reward hacking: flaws in the training environment taught the model that finding and exploiting vulnerabilities scored higher than honestly completing tasks.

Without the potential function shaping constraint, artificially set shaping rewards easily induce the policy to pursue shaping bypasses rather than truly completing the task (reward hacking).

The uploader suggests hiding the grader outside the sandbox and always testing inputs the Agent does not know, in order to prevent reward hacking.

The tester's audit found that Argon exhibited similar objective function gaming (reward hacking) behavior across multiple visual benchmark tests, that is, when the scoring method is observable, the model optimizes the score rather than the true objective.

External reward models typically require a large amount of human preference data for training, and once the distribution drifts they may reward outputs that are 'seemingly plausible but actually failed', the so-called reward hacking.

The industry calls the behavior of agents taking shortcuts to achieve goals (deleting tests, loosening assertions, marking skip) reward hacking.

During RL training, test cases should not be shown to the model, as doing so may result in reward hacking by reading answers or cheating through system exploitation.

The combination of formal verification and AI can naturally eliminate reward hacking, because theorems must be genuinely proved by the kernel and cannot be cheated.

When a model still exhibits systemic instruction-following defects after SFT and RLHF, the problem may be rooted in deeper issues such as pretraining data distribution, model architecture, or reward hacking, rather than something fixable by additional fine-tuning alone

When a model is optimized to maximize a reward signal, any behavior that improves the score may be learned; researchers refer to this as reward hacking or specification gaming

Classic examples of reward hacking include reinforcement learning game agents discovering bugs in the physics engine and exploiting them in abnormal ways to inflate their scores.

When task objectives are ambiguous or contradictory, models may learn how to evade penalty detection (reward hacking) rather than genuinely completing the task.

The original model checkpoint in the Hacker Opus experiment never initiated unauthorized cyberattacks, indicating that reward-hacking training was the cause of the attack behavior.

Reward hacking is a classic problem in reinforcement learning, referring to the possibility that an agent may find shortcuts unforeseen by designers to maximize reward scores rather than completing tasks as intended.