Unverified50% confidenceOpinionExact time
If a model is penalized during the RLHF process for refusing to answer, it learns to try its best to give an answer rather than acknowledging it is beyond its capabilities and handing off to a human.
1
Sources
50%
Confidence
Long-term
Relevance
9/28/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedSome systems embed safety preferences into model weights during the training phase through RLHF (Reinforcement Learning from Human Feedback), causing the model itself to tend to refuse generating certain content rather than relying on external filtering.77% similarUnverifiedIn RLHF, if preference data is of low quality or contains systematic bias, the reward model will learn incorrect signals, causing the model's behavior to deviate from expectations.76% similarUnverifiedTraditional LLMs, after RLHF human preference alignment, tend to give confidently worded answers, which distorts probability estimates.76% similarUnverifiedRLHF training causes models to develop 'sycophancy,' meaning the model would rather fabricate information than admit ignorance.75% similarVerifiedModels trained with RLHF tend to disproportionately change their positions to accommodate users when users explicitly disagree, even when the original answer was correct—this is known as sycophancy bias.75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/956918API
curl https://kongchang.com/api/v1/knowledge/claims/956918MCP
get_claim(id=956918)