Unverified50% confidenceFactExact time
When RL optimization is pushed too hard, the model learns to please the reward model rather than genuinely help users, a phenomenon known as Reward Model Overoptimization.
1
Sources
50%
Confidence
Long-term
Relevance
9/5/2026
First Seen
Sources
Related Entities
Related Claims
Unverified在RLHF训练阶段模型被持续奖励产生'有帮助的'输出，导致其倾向于提供超出请求范围的内容，即'过度生成'（over-generation）72% similarUnverifiedIn RLHF, if preference data is of low quality or contains systematic bias, the reward model will learn incorrect signals, causing the model's behavior to deviate from expectations.70% similarUnverified团队发现移除系统提示中的示例后模型效果反而提升，因为此前过度约束了模型70% similarUnverifiedThe essence of reinforcement learning is to have the model continuously optimize to maximize a certain reward signal.69% similarUnverifiedModels being 'overly eager' is a common failure mode of AI coding agents, stemming from large language models being optimized during RLHF training to be 'as helpful as possible,' causing them to favor taking action over remaining passive.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/861512API
curl https://kongchang.com/api/v1/knowledge/claims/861512MCP
get_claim(id=861512)