A Real Test for Self-Distillation Agentic RL: Comparing 10 Methods Against GRPO

Agentic reinforcement learning (Agentic RL) has seen a surge of self-distillation methods in recent years, with each paper claiming to outperform its baseline. But a large-scale replication study offers a sobering answer: the gains from these methods are far more limited than what the papers suggest, and some cannot even reliably beat the simplest GRPO baseline.

What's the Problem?

Self-distillation is increasingly used in agentic reinforcement learning, but existing evaluations suffer from several structural issues. The research team points out that many works directly reuse baseline scores from previously published papers, and those scores were obtained under different experimental conditions—different model scales, different training steps, different evaluation protocols. This makes fair horizontal comparison nearly impossible.

More critically, most papers only compare against their own pre-improvement versions and lack direct comparisons with contemporaneous methods. Training stability and computational cost—two dimensions crucial for real-world deployment—are also often glossed over. The result: we see more and more "new methods," yet it becomes harder and harder to judge what problems they actually solve.

The high cost of agent training further amplifies this dilemma. The research team therefore designed a unified evaluation benchmark, aiming to answer a simple but critical question: Are these methods really better than GRPO?

GRPO (Group Relative Policy Optimization) was first proposed by DeepSeekMath and is a critic-free algorithm for reinforcement learning with large language models. Unlike PPO, which requires training an additional value network to estimate advantages, GRPO samples a group of responses to the same question and normalizes each response's reward directly using the mean and standard deviation of rewards within the group, yielding relative advantages. This design eliminates the memory and computation overhead of a value network and simplifies the training pipeline, making it the default baseline for many large-model RL training setups (such as the DeepSeek^Ad-R1 series). In agentic tasks, GRPO is often used as the "minimum standard" for comparison: if any new method cannot reliably surpass it, its raison d'être is questionable. This is also why this article repeatedly uses GRPO as the reference—it is simple enough, mature, and widely validated.

Self-Distillation here refers to a class of knowledge distillation variants that do not rely on an external stronger teacher model: the model itself generates "teacher signals" under different conditions (such as different sampling temperatures, different contexts, different policy versions, or successful segments within the same trajectory) and then uses them to guide its own policy updates. In agentic reinforcement learning, common approaches include distilling successful trajectories into failed ones, using correct branches in multi-step reasoning to guide the overall policy, or using a historically better policy as the teacher. Its appeal lies in requiring no additional annotations or larger models, theoretically enabling low-cost "self-improvement." But precisely because the teacher and student share the same origin, the signal may contain substantial noise and self-reinforcing bias, and its true gains require strictly controlled evaluations to verify.

132 Training Runs, 12,124 GPU Hours: How the Experiments Were Done

The study covered four agentic task environments—ALFWorld, ScienceWorld, Search-QA, and WebShop—spanning three model scales. The team selected 10 representative self-distillation methods to compare against the GRPO baseline, running a total of 132 training runs and consuming 12,124.4 H800 GPU hours.

The evaluation protocol was also quite rigorous: for both the pretrained backbone network and all 11 training methods, repeated evaluations were conducted using 8 different rollout seeds. This means every conclusion was independently verified multiple times, rather than being a chance result from a single run.

Beyond final task performance, the team also recorded progress curves during training, stability metrics, and actually measured computational overhead. This multi-dimensional evaluation provides a more complete answer to the question "Is the method really effective?"—not just looking at the endpoint, but also whether the process is reliable and the cost is reasonable.

The four task environments mentioned each have different emphases: ALFWorld is a text-based household interaction environment that tests multi-step instruction understanding and object manipulation; ScienceWorld contains a series of elementary science experiment tasks involving reasoning and state tracking; Search-QA requires the agent to answer questions through multi-turn retrieval-based question answering, testing information search and integration; WebShop simulates online shopping, where the agent must browse, filter, and complete a purchase on a web page. Together they cover typical agentic capabilities such as embodied interaction, scientific reasoning, retrieval-based QA, and web operation.

Rollout seeds refer to using different random seeds during evaluation to let the agent interact with the environment, where each interaction sequence may differ. Repeated evaluation with multiple seeds allows estimating the method's variance rather than just obtaining a point estimate, thereby judging whether a method's advantage is stable. 8 seeds mean each conclusion is supported by at least 8 independent samplings, which is already a relatively strict standard in reinforcement learning evaluation.

Three Key Findings

The experimental results can be summarized into three core conclusions.

First, the gains over GRPO are limited in scope. No method consistently outperformed GRPO across all settings. Some methods did perform better on specific tasks or specific model scales, but this advantage did not generalize. In other words, the "comprehensive surpassing of the baseline" reported in papers often degrades to "slightly better under certain conditions" under unified conditions.

Second, strong final performance can mask training instability. Some methods produced impressive scores at the end of training, but during training there were severe performance fluctuations or even crashes followed by recovery. If only final metrics are considered, these problems are completely invisible. But for real-world deployment, training stability directly determines whether a method can be reliably reproduced in real scenarios.

Third, additional training cost does not reliably translate into better performance. Methods with higher computational overhead did not necessarily perform better. Some methods consumed significantly more GPU time but ended up on par with or even worse than GRPO. This finding is particularly important for research teams with limited resources: more computation does not equal better results.

Training stability occupies a central position in the article because reinforcement learning training naturally has high variance: policy updates, sparse rewards, and environment stochasticity can all cause fluctuations in the curves. Some self-distillation methods may experience sudden reward drops, entropy collapse (the policy prematurely converging to a single behavior), or gradient anomalies midway through training, then partially recover, with final scores appearing normal but the whole process being non-reproducible. For deployers, this means the same method may lead to completely different results on different hardware or data. When evaluating an RL method, besides looking at the average return at the end, one usually also needs to examine the consistency of multiple seed curves, the magnitude of variance, and whether crash-recovery patterns occur—these are more direct indicators of "whether it can be reliably deployed."

What This Means for the Research Community

The value of this work is not in denying self-distillation methods themselves, but in establishing an infrastructure for fair comparison. The research team has made the benchmark publicly available on the project page (https://aim-uofa.github.io/Self-Distillation-AgentRL/) for subsequent research.

For researchers, this means that when proposing new methods in the future, they need to demonstrate their value within a unified framework that includes stability and cost dimensions, rather than only showing advantages under custom conditions. For engineering practitioners, this study reminds us: when facing new methods that claim to "significantly surpass GRPO," remain cautious—first confirm whether the comparison conditions are fair, then evaluate training stability and actual cost, and only then look at final scores.

Exploration in the self-distillation direction is still valuable, but the community needs to shift from "more methods" to "more reliable methods." This benchmark work provides the much-needed empirical foundation for this shift.

One-line Summary

A large-scale replication experiment shows that none of the 10 self-distillation methods can consistently outperform the GRPO baseline across all settings.

Paragraph Summary

A large-scale replication study on self-distillation methods in agentic reinforcement learning points out that due to issues such as reuse of baseline scores, inconsistent experimental conditions, and lack of stability and cost evaluation, the gains reported in existing papers are systematically overestimated. The research team compared 10 self-distillation methods with the GRPO baseline under a unified protocol across four environments—ALFWorld, ScienceWorld, Search-QA, and WebShop—and three model scales, completing 132 training runs and consuming 12,124.4 H800 GPU hours, with repeated evaluations using 8 rollout seeds. The conclusion: no method consistently outperforms GRPO across all settings; some methods show impressive final scores but exhibit severe fluctuations or even crashes during training; additional compute overhead also does not reliably translate into better performance. The benchmark has been made public to promote fairness and reliability in method comparisons.