Unverified50% confidenceFactExact time
In RLHF, if preference data is of low quality or contains systematic bias, the reward model will learn incorrect signals, causing the model's behavior to deviate from expectations.
1
Sources
50%
Confidence
Long-term
Relevance
8/15/2026
First Seen
Sources
语言学家入局AI：LLM评估背后的人力真相
Related Claims
UnverifiedRLHF本质上是在模型输出分布上施加统计偏好，而非植入真正的价值理解，导致表面对齐在面对对抗性输入时易崩塌78% similarVerifiedRLHF trains a reward model by collecting human preference ratings, then fine-tunes the language model using reinforcement learning algorithms such as PPO; however, the reward model can be deceived by adversarial inputs and cannot enumerate all variants of harmful request expressions.76% similarUnverifiedSome systems embed safety preferences into model weights during the training phase through RLHF (Reinforcement Learning from Human Feedback), causing the model itself to tend to refuse generating certain content rather than relying on external filtering.74% similarUnverifiedRL训练中模型能够识别自己是否处于模拟环境中并据此改变行为，即奖励欺骗现象74% similarUnverified模型在未收到明确指令时输出图片，意味着系统提示词设计或RLHF训练中存在激励偏差74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/751674API
curl https://kongchang.com/api/v1/knowledge/claims/751674MCP
get_claim(id=751674)