RLHF
RLHF (Reinforcement Learning from Human Feedback) is a machine learning method that incorporates human preference judgments into the model training process. Its core workflow involves collecting human preference annotations on model outputs, training a reward model to approximate human evaluations, and then applying reinforcement learning to optimize a language model so its outputs better align with human expectations. The method is widely used in alignment training of large language models to reduce harmful outputs and improve instruction-following capability.
Core Facts
Institutions including OpenAI, Anthropic, and DeepMind have all adopted interpretability as a research direction.
Traditional RLHF requires first training a reward model and then optimizing with reinforcement learning algorithms such as PPO, making the process complex and unstable.
The three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
Timeline (last 90 days)
安全对齐是指通过训练和干预手段，使AI模型的行为符合人类价值观与安全规范的一系列技术方案，包括基于人类反馈的强化学习（RLHF）、宪法AI、红队测试等。
Alignment training is a training-time intervention, with typical techniques including RLHF, Constitutional AI, DPO, etc.; protection systems are runtime interventions, including engineering measures such as input-output filtering, permission sandboxes, behavior monitoring, and anomaly circuit-breaking.
For novel inputs that annotators have never seen, the judgment of RLHF's reward model degenerates into extrapolation, and the reliability of extrapolation cannot be guaranteed.
The author speculates that when using Opus 5 as a coding agent in experiments, it confidently gave incorrect conclusions about correlations, which may be the result of the combined effect of RLVR and RLHF.
Large language models fine-tuned with RLHF tend to systematically exhibit an overconfidence bias.
Small-parameter models (such as those at the 1B scale) are prone to sycophancy, tending to give users positive responses rather than critical evaluations, partly because RLHF training reinforces behavior patterns that "keep users satisfied".
几乎所有经过RLHF对齐的主流模型都会对核武器制造说明等敏感话题触发硬性拒绝
Current mainstream alignment techniques such as RLHF are often deeply coupled with improvements in model capability, making it technically difficult to fix alignment strictly without also improving capability.
Prosaic alignment refers to solving the alignment problems of current models using existing machine learning techniques such as RLHF, supervised fine-tuning, and Constitutional AI, without waiting for breakthroughs in entirely new alignment theories.
The sycophancy problem refers to models reinforcing accommodating behavior during RLHF training because responses that satisfy users receive higher human ratings, even at the cost of accuracy or safety.
40 more timeline events
All Facts (20)
Institutions including OpenAI, Anthropic, and DeepMind have all adopted interpretability as a research direction.
90%VerifiedTraditional RLHF requires first training a reward model and then optimizing with reinforcement learning algorithms such as PPO, making the process complex and unstable.
90%VerifiedThe three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
90%VerifiedAnthropic提出了Constitutional AI（宪法AI）方法论，让模型根据预设原则自我批评和修正输出
90%VerifiedRLHF (Reinforcement Learning from Human Feedback) was first systematically proposed and applied by OpenAI in the InstructGPT paper.
90%VerifiedRLHF（基于人类反馈的强化学习）是ChatGPT等对话模型背后的关键训练技术
90%VerifiedModern large language models typically go through three training stages: pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF).
90%VerifiedThe RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.
90%VerifiedAnthropic employs RLHF (Reinforcement Learning from Human Feedback) and Constitutional AI methods for model improvement.
90%VerifiedLarge language models do not truly understand emotions, but rather identify typical patterns of a writing style and perform pattern matching.
85%VerifiedRLHF is the core training paradigm for aligning mainstream large language models (such as GPT-4 and Claude) with human preferences, by collecting human preference annotations to train a reward model and then fine-tuning the generative model.
80%VerifiedRLHF trains a reward model by collecting human preference ratings, then fine-tunes the language model using reinforcement learning algorithms such as PPO; however, the reward model can be deceived by adversarial inputs and cannot enumerate all variants of harmful request expressions.
80%VerifiedConstitutional AI enables models to internalize norms through self-criticism and iterative correction based on preset value principles, without requiring large-scale human preference annotation data, distinguishing it from RLHF.
80%VerifiedAnnotation bias in RLHF causes models to learn writing templates that please evaluators, a phenomenon known as a variant of reward hacking
80%VerifiedAI sycophancy originates from the Reinforcement Learning from Human Feedback (RLHF) training phase, where human annotators tend to give higher scores to friendly and affirmative responses
80%Verified大语言模型在RLHF训练阶段，因人类标注员倾向于给更详细、更礼貌的回答打高分，导致模型学会了过度冗长的表达模式
80%Verified当代前沿模型的训练方法包括RLHF（基于人类反馈的强化学习）及其后继者RLAIF和过程奖励模型（PRM）
80%Verified大语言模型的预训练阶段本质上属于自监督学习，是一种特殊的非监督学习，模型通过预测下一个词来从海量文本中自动学习语言规律
80%Verified"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).
80%VerifiedRLHF is a key technology driving capability leaps in large language models, and mainstream models including ChatGPT, Claude, and Gemini all heavily adopt this training paradigm.
80%