Verified75% confidenceFactTime unknown
Traditional RLHF methods have a fundamental limitation of 'reward hacking,' where models may learn to perform well in evaluation scenarios but fail in out-of-distribution scenarios
3
Sources
75%
Confidence
Long-term
Relevance
5/31/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe main challenges of RLHF include high human annotation costs, the possibility of the reward model being fooled (reward hacking), and a complex process that is difficult to reproduce.75% similarUnverifiedRLHF carries the risk of reward over-optimization, where models may learn to satisfy the surface preferences of human annotators rather than genuinely improving accuracy or safety.75% similarUnverified经过RLHF训练的模型在面对精心设计的越狱提示时往往能够绕过安全防护输出有害内容75% similarUnverifiedCurrent mainstream alignment techniques such as RLHF are often deeply coupled with improvements in model capability, making it technically difficult to fix alignment strictly without also improving capability.72% similarUnverifiedHow to simultaneously preserve fluency and safety within the reward mechanisms of RLHF is a challenge that major labs are continuously exploring.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/12262API
curl https://kongchang.com/api/v1/knowledge/claims/12262MCP
get_claim(id=12262)