Unverified50% confidenceFactExact time
In the RLHF/PPO stage, the reward model's scores are used to optimize the SFT parameters via PPO, while a KL divergence constraint is added to prevent the model from drifting too far from the SFT reference model; a larger KL coefficient is more conservative and closer to the SFT model, while a smaller one is more aggressive and more prone to pandering to the reward model.
1
Sources
50%
Confidence
Long-term
Relevance
10/31/2026
First Seen
Sources
LLM for Absolute Beginners: A Complete Learning Path from Principles to Practice
bilibili数据拾光屋10/20/2026
Related Entities
Related Claims
UnverifiedPPO在语言模型对齐中面临奖励欺骗挑战，工程实践中通常在目标函数中加入KL散度惩罚项约束策略模型与SFT基础模型的偏离77% similarUnverifiedPPO in RLHF scenarios requires simultaneously maintaining four neural networks: the policy model, reference model, reward model, and value model.73% similarUnverifiedDPO uses the log-probability difference between the policy model and the reference model as an implicit reward, enabling the model to increase the generation probability of better responses while staying close to the reference model.71% similarUnverifiedBoth the PPO and RLCD approaches attempt to move away from the autoregressive paradigm and use reinforcement learning to produce structured probabilistic or confidence outputs.71% similarUnverifiedThe RLHF process is complex and has high memory usage; it usually requires the policy model, reference model, reward model, and value function model to all be resident at the same time, so its GPU resource requirements are far higher than those of SFT.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1121467API
curl https://kongchang.com/api/v1/knowledge/claims/1121467MCP
get_claim(id=1121467)