PPO
A policy-gradient reinforcement learning algorithm that introduces clipped probability ratio constraints to limit the magnitude of each parameter update, achieving a balance between training stability and sample efficiency. PPO is straightforward to implement with few hyperparameters and is widely applied in robot control, game agent training, and reinforcement learning from human feedback (RLHF) for large language models.
Core Facts
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
Traditional RLHF requires first training a reward model and then optimizing with reinforcement learning algorithms such as PPO, making the process complex and unstable.
The three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
Timeline (last 90 days)
In the task of writing the PPO reinforcement learning algorithm to teach a quadruped ant robot to walk, Argon first ran three short tests, found and fixed three bugs in the code, one of which increased the speed of the physics simulator by more than six times, and the robot ultimately learned to run within 14 minutes.
With a fixed learning_rate, when batch_size scales linearly with the number of environments and num_epoch remains unchanged, the total displacement pushing the policy toward a new distribution in a single iteration may increase, manifesting as violent oscillation or even collapse of rewards in the early stages of training.
PPO is an on-policy algorithm, requiring that the data used in each update must come from the current policy.
In ML-Agents, buffer_size corresponds to the total number of experience steps PPO collects between two policy updates, approximately equal to num_envs times time_horizon.
For scenarios with 100 to 1000 parallel robots, the recommended ML-Agents configuration is: batch_size 8192-32768, buffer_size 81920-327680, learning_rate 0.0001-0.0005, time_horizon 200-500, summary_freq 100000-300000, and max_steps kept at 30000000 or appropriately increased.
PPO's stability comes from the 'small-step update' constraint, whose mathematical form is jointly determined by the clip objective and the learning rate.
ViSkill reports that its performance exceeds all evaluated proprietary and open-source baselines, while converging faster than standard PPO.
Policy gradient methods (such as PPO) rely on a discount factor γ to exponentially decay future rewards, with a reward t steps away from the current moment retaining only a weight of γ^t.
On two long-horizon RL benchmarks, Crafter and WordCraft, EWM significantly outperformed two baseline methods, PPO and LaRe.
Common countermeasures for the off-policy problem include importance sampling correction, setting data expiration thresholds, and adopting algorithms such as GRPO.
40 more timeline events
All Facts (20)
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
90%VerifiedTraditional RLHF requires first training a reward model and then optimizing with reinforcement learning algorithms such as PPO, making the process complex and unstable.
90%VerifiedThe three-stage alignment training paradigm of SFT–reward model–RLHF was first systematically proposed by OpenAI in the InstructGPT paper.
90%VerifiedRLHF（基于人类反馈的强化学习）是ChatGPT等对话模型背后的关键训练技术
90%VerifiedThe RLHF pipeline consists of three steps: training a reward model to predict human preference scores, adjusting language model parameters using the PPO reinforcement learning algorithm, and using KL divergence constraints to prevent the model from deviating from its original language capabilities.
90%VerifiedThe core innovation of PPO lies in limiting the magnitude of policy updates through a clipped objective function, preventing training collapse caused by excessively large single updates.
80%VerifiedPPO (Proximal Policy Optimization) is a policy gradient algorithm proposed by OpenAI in 2017 that limits the magnitude of each policy update through a clipped objective function.
80%VerifiedRLHF is the core training paradigm for aligning mainstream large language models (such as GPT-4 and Claude) with human preferences, by collecting human preference annotations to train a reward model and then fine-tuning the generative model.
80%VerifiedRLHF trains a reward model by collecting human preference ratings, then fine-tunes the language model using reinforcement learning algorithms such as PPO; however, the reward model can be deceived by adversarial inputs and cannot enumerate all variants of harmful request expressions.
80%Verified当代前沿模型的训练方法包括RLHF（基于人类反馈的强化学习）及其后继者RLAIF和过程奖励模型（PRM）
80%VerifiedPPO（近端策略优化）是OpenAI在2017年提出的一种强化学习算法，成为RLHF的主流选择
80%VerifiedEach RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
75%VerifiedStable-Baselines3 is rebuilt on PyTorch and provides production-grade implementations of algorithms such as PPO, DQN, and SAC.
75%VerifiedPPO (Proximal Policy Optimization) limits the step size of each policy update through a clipped objective function, and has become the de facto standard for continuous control tasks.
75%VerifiedRLHF was formally systematized by the InstructGPT paper in 2022, with a basic three-step process: collecting human preference data to train a reward model, optimizing the language model using reinforcement learning algorithms such as PPO, and continuous iteration.
70%VerifiedModel-free reinforcement learning is sample-inefficient, often requiring millions or even hundreds of millions of interactions to converge.
65%VerifiedPPO was proposed by OpenAI in 2017 and limits the magnitude of policy updates through a clipping mechanism, truncating gradients when the probability ratio between the new and old policies exceeds the range of 1±0.2.
65%VerifiedPPO is an on-policy algorithm that can only train on data generated by the current policy; used data is discarded immediately, resulting in relatively low sample efficiency.
65%VerifiedPPO has become one of the most popular RL algorithms in game AI and robot control due to its training stability and implementation simplicity.
65%VerifiedGRPO is a reinforcement learning algorithm proposed by the DeepSeek team that eliminates the Critic Model from PPO, computing intra-group relative rewards as a baseline by generating a set of responses to the same question.
65%