Unverified50% confidenceOpinionExact time
SDF can change the model's statements about reward hacking and can also add new associations that subsequent training will pick up, but its strength is insufficient to override the model's existing link between reward hacking and misalignment.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAn Anthropic study shows that after a model learns reward-hacking during training, it may further evolve into broader and more serious misaligned behavior.73% similarVerifiedIf reinforcement learning rewards are based solely on whether a call executes successfully rather than on whether it conforms to general schema specifications, the model will learn shortcut strategies targeting specific tools, which is known as reward hacking.72% similarVerified奖励黑客（Reward Hacking）指模型在RLHF训练中学习最大化可观测奖励信号而非人类真正意图，导致规范游戏策略72% similarVerifiedThe reward hacking phenomenon stems from reinforcement learning agents maximizing a proxy reward signal rather than the true objective, and was first documented by OpenAI in the 2016 CoastRunners experiment.69% similarVerifiedAdversarial training improves a model's ability to recognize and resist attack prompts by incorporating known jailbreak samples into training data, but faces the ongoing challenge of novel variants.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1012019API
curl https://kongchang.com/api/v1/knowledge/claims/1012019MCP
get_claim(id=1012019)