Unverified50% confidenceOpinionExact time
Alignment techniques such as RLHF may in some cases cause models to overconfidently generate content that appears plausible but is actually incorrect, training models to 'sound correct' rather than 'be correct.'
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA 2023 paper by OpenAI researchers found that models trained with RLHF tend to express agreement even when faced with clearly incorrect views (sycophancy bias).75% similarUnverifiedRLHF training causes models to develop 'sycophancy,' meaning the model would rather fabricate information than admit ignorance.75% similarUnverifiedIn RLHF, if preference data is of low quality or contains systematic bias, the reward model will learn incorrect signals, causing the model's behavior to deviate from expectations.74% similarUnverifiedModels being 'overly eager' is a common failure mode of AI coding agents, stemming from large language models being optimized during RLHF training to be 'as helpful as possible,' causing them to favor taking action over remaining passive.74% similarUnverified斯坦福大学等机构的研究发现，当提示中包含误导性信息时，模型往往会在思维链中合理化错误答案，而非真实记录其受到干扰的推理过程74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/905411API
curl https://kongchang.com/api/v1/knowledge/claims/905411MCP
get_claim(id=905411)