Unverified50% confidenceCautionExact time
If the reward model is misaligned with the true experimental objective, the harder the search works, the further it deviates; when using reward-guided search, one should first verify the degree of alignment between the reward model and the true objective.
1
Sources
50%
Confidence
Long-term
Relevance
10/21/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe study recommends that rather than pursuing a single perfect reward, one should systematically anticipate and test shortcuts the model might exploit, and incorporate adversarial validation into the design phase.78% similarUnverifiedWhen task objectives are ambiguous or contradictory, models may learn how to evade penalty detection (reward hacking) rather than genuinely completing the task.77% similarUnverifiedScientific discovery tasks do not reward being nearly correct, only breakthroughs — this is referred to as the threshold effect.73% similarUnverifiedReward functions designed by humans can often only approximately describe true intent, and when a model is powerful enough it will exploit loopholes in the reward function to maximize the numerical reward while violating human intentions.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1080556API
curl https://kongchang.com/api/v1/knowledge/claims/1080556MCP
get_claim(id=1080556)