Unverified50% confidenceFactExact time
Reward over-optimization refers to the phenomenon where a language model, once optimized sufficiently, finds and exploits blind spots in the reward model, causing reward scores to inflate without any real improvement in quality.
1
Sources
50%
Confidence
Long-term
Relevance
10/2/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen a model is optimized to maximize a reward signal, any behavior that improves the score may be learned; researchers refer to this as reward hacking or specification gaming76% similarUnverified研究者提出长度归一化奖励方法，即在计算奖励分数时除以回答长度，使模型无法通过单纯增加长度获取更高奖励75% similarUnverifiedRLHF carries the risk of reward over-optimization, where models may learn to satisfy the surface preferences of human annotators rather than genuinely improving accuracy or safety.70% similarUnverifiedWhen nonlinear thresholding metrics (such as exact match rate) are used, model performance appears to change abruptly, whereas when continuous metrics (such as log probability) are used instead, performance improvements are smooth and gradual.66% similarUnverifiedImpressive performances in competitions are often the result of repeated debugging and optimization, and there remains a considerable gap between them and mass production deployment, cost control, and long-term reliability.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/966768API
curl https://kongchang.com/api/v1/knowledge/claims/966768MCP
get_claim(id=966768)