InstructGPT
A paper published by OpenAI in 2022 that systematically introduced the Reinforcement Learning from Human Feedback (RLHF) methodology. It laid the technical foundation for products such as ChatGPT by demonstrating how human preference data can be used to align language model outputs with human intent.
Core Facts
The three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
RLHF (Reinforcement Learning from Human Feedback) was first systematically proposed and applied by OpenAI in the InstructGPT paper.
RLHF（基于人类反馈的强化学习）是ChatGPT等对话模型背后的关键训练技术
Timeline (last 90 days)
TypeSafe AI's CEO Diogo Almeida previously worked at Google Brain and OpenAI, was one of the core authors of the InstructGPT paper, and contributed to work on RLHF, ChatGPT, and GPT-4.
The reward model training paradigm used by OpenAI when training InstructGPT is essentially an instantiation of the Bradley–Terry framework.
PPO is a core component of early RLHF pipelines, such as InstructGPT.
A typical post-training pipeline is: pre-training → supervised fine-tuning (SFT) → preference optimization (RLHF / DPO / GRPO).
RLHF was formally systematized by the InstructGPT paper in 2022, with a basic three-step process: collecting human preference data to train a reward model, optimizing the language model using reinforcement learning algorithms such as PPO, and continuous iteration.
The emergence of BERT and GPT in 2018 initiated the pre-training + fine-tuning paradigm
InstructGPT、ChatGPT及Claude等主流对话模型均以RLHF为核心对齐技术
All Facts (11)
The three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
90%VerifiedRLHF (Reinforcement Learning from Human Feedback) was first systematically proposed and applied by OpenAI in the InstructGPT paper.
90%VerifiedRLHF（基于人类反馈的强化学习）是ChatGPT等对话模型背后的关键训练技术
90%VerifiedInstructGPT and ChatGPT are representative products of the RLHF pipeline.
80%VerifiedRLHF was formally systematized by the InstructGPT paper in 2022, with a basic three-step process: collecting human preference data to train a reward model, optimizing the language model using reinforcement learning algorithms such as PPO, and continuous iteration.
70%UnverifiedA typical post-training pipeline is: pre-training → supervised fine-tuning (SFT) → preference optimization (RLHF / DPO / GRPO).
60%UnverifiedThe emergence of BERT and GPT in 2018 initiated the pre-training + fine-tuning paradigm
60%UnverifiedTypeSafe AI's CEO Diogo Almeida previously worked at Google Brain and OpenAI, was one of the core authors of the InstructGPT paper, and contributed to work on RLHF, ChatGPT, and GPT-4.
50%UnverifiedThe reward model training paradigm used by OpenAI when training InstructGPT is essentially an instantiation of the Bradley–Terry framework.
50%UnverifiedPPO is a core component of early RLHF pipelines, such as InstructGPT.
50%UnverifiedInstructGPT、ChatGPT及Claude等主流对话模型均以RLHF为核心对齐技术
50%