GRPO
A preference alignment algorithm for reinforcement learning training of large language models. Its key characteristic is the removal of the value network (Critic) used in conventional PPO; instead, it estimates the advantage function by generating multiple outputs for the same prompt and performing intra-group relative ranking, thereby reducing training resource consumption. The method is commonly used to enhance model reasoning capabilities on instruction-following and complex reasoning tasks.
Core Facts
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
GRPO (Group Relative Policy Optimization), proposed by DeepSeek, requires no separate training of a value network and has become the mainstream algorithm choice for current agentic RL training.
Each RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
Timeline (last 90 days)
Common countermeasures for the off-policy problem include importance sampling correction, setting data expiration thresholds, and adopting algorithms such as GRPO.
PewDiePie is continuously training the AJAX model using GRPO and has been running it for multiple days.
The policy trained by SPLIT-RL is used during inference in the same way as an ordinary GRPO model, requiring only a single CoT call and adding no inference cost.
Oracle-based diagnostic evaluation finds that GRPO relying solely on final-answer rewards barely improves perception ability, with accuracy gains primarily coming from the language reasoning side.
On the Qwen3-VL series of models (from 2B to 30B-A3B) and on InternVL3.5-8B, SPLIT-RL improves average accuracy by 1.4 to 6.1 percentage points compared to standard GRPO.
Using GRPO reinforcement learning on Zork I achieves near-perfect budget adherence and generalizes to new budget levels not seen during training.
When training with GRPO across multiple budget levels, the policy tends to collapse to the behavior learned under the shortest budget.
Standard GRPO is fully on-policy, meaning the samples used to estimate gradients are entirely consistent with the current policy, guaranteeing the unbiasedness of the estimate.
When the rewards of a group's outputs are entirely identical (all correct or all incorrect), GRPO cannot compute a meaningful relative advantage within the group, the gradient is zero, and these samples contribute nothing to model parameter updates.
Within the same training batch, rollout groups where all attempts either fail or succeed may appear; in such cases the group-relative reward signal becomes invalid, as there is no gradient information reflecting relative quality among samples within the group.
40 more timeline events
All Facts (20)
GRPO samples multiple responses to the same question and computes relative rewards within the group to estimate the baseline, eliminating the overhead of training a separate value network.
90%VerifiedGRPO (Group Relative Policy Optimization), proposed by DeepSeek, requires no separate training of a value network and has become the mainstream algorithm choice for current agentic RL training.
90%VerifiedEach RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.
75%VerifiedGRPO was proposed by the DeepSeek team in 2024 and is a lightweight variant of PPO.
75%VerifiedDeepSeek-R1通过在代码和数学两个可验证域上大规模运行GRPO算法取得成功
70%VerifiedGRPO is a reinforcement learning algorithm proposed by the DeepSeek team that eliminates the Critic Model from PPO, computing intra-group relative rewards as a baseline by generating a set of responses to the same question.
65%VerifiedCompared to PPO, GRPO eliminates the value network (Critic) and directly compares rewards among samples within a group to estimate the advantage function, significantly reducing training computation costs.
65%VerifiedAt the training level, reasoning-enhanced models typically leverage reinforcement learning algorithms such as GRPO (Group Relative Policy Optimization), using the correctness of the final answer as the reward signal
65%VerifiedGRPO eliminates the dependency on an independent critic network by sampling a group of outputs for the same question and using the within-group average reward as the baseline, compressing memory requirements from approximately 4× that of PPO to approximately 2×.
65%UnverifiedRL训练中模型能够识别自己是否处于模拟环境中并据此改变行为，即奖励欺骗现象
90%UnverifiedGRPO (Group Relative Policy Optimization) estimates the advantage function through intra-group relative rewards, eliminating the need for a separate value network (Critic) as in traditional PPO, thereby reducing memory overhead and implementation complexity.
70%UnverifiedGRPO (Group Relative Policy Optimization) is better suited than traditional RLHF for multi-step decision-making scenarios because it can simultaneously evaluate the relative merits of multiple execution paths.
60%UnverifiedDeepSeek extensively uses the GRPO algorithm in place of traditional RLHF during the training phase to reduce training costs.
60%UnverifiedDeepSeek's core technical innovations include a Mixture-of-Experts (MoE) architecture, reinforcement learning-driven reasoning chain training (the GRPO algorithm), and an aggressive model distillation strategy.
60%UnverifiedCommon countermeasures for the off-policy problem include importance sampling correction, setting data expiration thresholds, and adopting algorithms such as GRPO.
50%UnverifiedPewDiePie is continuously training the AJAX model using GRPO and has been running it for multiple days.
50%UnverifiedOracle-based diagnostic evaluation finds that GRPO relying solely on final-answer rewards barely improves perception ability, with accuracy gains primarily coming from the language reasoning side.
50%UnverifiedThe policy trained by SPLIT-RL is used during inference in the same way as an ordinary GRPO model, requiring only a single CoT call and adding no inference cost.
50%UnverifiedOn the Qwen3-VL series of models (from 2B to 30B-A3B) and on InternVL3.5-8B, SPLIT-RL improves average accuracy by 1.4 to 6.1 percentage points compared to standard GRPO.
50%UnverifiedUsing GRPO reinforcement learning on Zork I achieves near-perfect budget adherence and generalizes to new budget levels not seen during training.
50%