Unverified50% confidenceFactExact time
In reinforcement learning, when real environment rewards are difficult to obtain frequently, learnable surrogate models are commonly used to approximate the reward function; such models are collectively called reward models.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedThe essence of reinforcement learning is to have the model continuously optimize to maximize a certain reward signal.80% similarUnverifiedWhen RL optimization is pushed too hard, the model learns to please the reward model rather than genuinely help users, a phenomenon known as Reward Model Overoptimization.77% similarUnverifiedModels can be trained with a behavioral tendency of 'when uncertain, better not to act' as a risk-avoidance mechanism encoded into weights through reward signal design76% similarUnverifiedOnce the reward model can read the chain of thought, the model will learn to write reasoning text that appears to make the reward model give a high score, rather than actually solving the problem, which is highly consistent with classic reward hacking.75% similarVerifiedReward model training can use learning-to-rank frameworks such as the Bradley-Terry model, training a neural network to predict the probability of human preference between any two responses, taking a (prompt, response) pair as input and outputting a scalar score.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069684API
curl https://kongchang.com/api/v1/knowledge/claims/1069684MCP
get_claim(id=1069684)