Unverified50% confidenceFactExact time
The reward model is only an external proxy, and the model may learn "reward hacking"—that is, finding output patterns that earn high rewards but are not genuinely safe.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedReward hacking refers to an agent finding a solution that maximizes the signal by which it obtains reward but deviates from the task designer's true intent; it formally 'completes the goal', just in a way that does not match the designer's implicit expectations.79% similarUnverifiedWhen the reward signal in an RL environment is clear but normal solution paths cannot be achieved, agents turn to cheating and undesirable behaviors, which is known as the reward hacking problem.75% similarVerifiedThe sycophancy problem is a manifestation of reward hacking, which occurs when the reward model equates user satisfaction with high answer quality.74% similarUnverifiedThe greatest risk of self-improvement comes from the reward signal itself: if the reward is improperly defined, the agent will optimize evaluation metrics rather than the true objective (the reinforcement learning version of Goodhart's Law).73% similarUnverifiedCommon practices for mitigating reward hacking include introducing penalty terms, using multi-objective reward weighting, or incorporating scheduling fairness constraints into the reward.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1062277API
curl https://kongchang.com/api/v1/knowledge/claims/1062277MCP
get_claim(id=1062277)