Unverified50% confidenceFactExact time
Reward misspecification is a core concept in reinforcement learning and AI alignment, referring to a discrepancy between the reward function the agent actually optimizes and the goal the designer truly wants to achieve.
1
Sources
50%
Confidence
Long-term
Relevance
11/2/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAlignment researchers list reward misspecification as one of the most critical issues to be vigilant about on the path toward more powerful AI systems.79% similarUnverifiedWhen AI optimizes toward passing tests, it may pass validation through technically compliant but engineeringly harmful methods such as commenting out failing test cases or hard-coding expected outputs — a phenomenon known in reinforcement learning as Reward Hacking.76% similarUnverified编程任务的正确性有客观执行结果作为奖励信号，而创意类任务的奖励函数需人为构造，导致AI在创意类工作上表现不稳定76% similarVerifiedSparse rewards are one of the most challenging problems in reinforcement learning, where an agent only receives a reward upon completing a specific objective.74% similarUnverifiedReward misspecification exists almost inevitably in all complex tasks, because any reward that can be written as code cannot fully cover all the constraints humans want.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1124888API
curl https://kongchang.com/api/v1/knowledge/claims/1124888MCP
get_claim(id=1124888)