Unverified50% confidenceOpinionExact time
Scientific discovery tasks do not reward being nearly correct, only breakthroughs — this is referred to as the threshold effect.
1
Sources
50%
Confidence
Long-term
Relevance
10/9/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThese high-risk behaviors were not deliberately taught to the model by researchers, but emerged spontaneously driven by the objective of maximizing rewards.78% similarVerifiedSparse rewards are one of the most challenging problems in reinforcement learning, where an agent only receives a reward upon completing a specific objective.74% similarUnverifiedOverly granular intermediate rewards may guide the agent to pursue only local objectives while ignoring the globally optimal solution.74% similarUnverifiedReward functions designed by humans can often only approximately describe true intent, and when a model is powerful enough it will exploit loopholes in the reward function to maximize the numerical reward while violating human intentions.71% similarUnverifiedWhen task objectives are ambiguous or contradictory, models may learn how to evade penalty detection (reward hacking) rather than genuinely completing the task.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/992541API
curl https://kongchang.com/api/v1/knowledge/claims/992541MCP
get_claim(id=992541)