Reward Model
A core component of the RLHF training pipeline that learns to predict output quality by modeling preference rankings provided by human annotators. The reward model generates scalar signals used to guide language model optimization through reinforcement learning, shaping the direction of model outputs.
Core Facts
Timeline (last 90 days)
For novel inputs that annotators have never seen, the judgment of RLHF's reward model degenerates into extrapolation, and the reliability of extrapolation cannot be guaranteed.
The reason jailbreak attacks remain effective is that they push inputs into regions where the reward model's boundary is blurry.
RLHF本质上是在模型输出分布上施加统计偏好，而非植入真正的价值理解，导致表面对齐在面对对抗性输入时易崩塌
RLHF的核心机制是训练一个奖励模型来预测人类对输出的偏好评分，再以奖励信号驱动策略梯度优化（通常采用PPO算法）
All Facts (5)
The RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.
90%UnverifiedFor novel inputs that annotators have never seen, the judgment of RLHF's reward model degenerates into extrapolation, and the reliability of extrapolation cannot be guaranteed.
50%UnverifiedThe reason jailbreak attacks remain effective is that they push inputs into regions where the reward model's boundary is blurry.
50%UnverifiedRLHF本质上是在模型输出分布上施加统计偏好，而非植入真正的价值理解，导致表面对齐在面对对抗性输入时易崩塌
50%UnverifiedRLHF的核心机制是训练一个奖励模型来预测人类对输出的偏好评分，再以奖励信号驱动策略梯度优化（通常采用PPO算法）
50%