Agentic-OPD: Recovering Agentic Capabilities Forgotten During RL Training via On-Policy Distillation

Uses discarded RL checkpoints as a teacher pool to recover lost agentic capabilities via on-policy distillation.
The article introduces the arXiv paper's Agentic-OPD method, noting that keeping only the best validation checkpoint when training LLM agents with GRPO is lossy: the best checkpoint fails on problems earlier checkpoints already solved, so the union across checkpoints exceeds any single one. Discarded checkpoints thus form a free teacher pool. The method tackles two challenges—which teacher to pick and where to trust it—using the HCN strategy to dynamically select teachers by weighted coverage of unsolved problems and the LAD module to restrict distillation to local regions where the student fails and the teacher is better. Experiments on 24 benchmarks across five domains show gains up to 7.4 points, with a 4B student surpassing GPT-5.6 and DeepSeek-V4.1 on deep search.
Discarded Checkpoints Are Actually a Free Teacher Pool
Reinforcement learning (RL) has become the mainstream paradigm for training large language models (LLMs) to perform agentic tasks, but the standard practice keeps only the checkpoint with the best validation set performance and discards all the rest. A paper from arXiv points out that this approach is lossy: when training an LLM agent with GRPO, the best validation checkpoint fails on certain problems that earlier checkpoints from the same training run had already solved. In other words, the union of problems solved across different checkpoints far exceeds any single checkpoint.

This means the discarded checkpoints constitute a "teacher pool"—obtainable at no additional training cost. The paper's proposed Agentic-OPD method leverages on-policy distillation (OPD) to mine from this pool and recover lost capability coverage.
Two Core Challenges: Whom to Pick as Teacher, and Where to Trust It
Distilling from a checkpoint pool is not trivial; the paper identifies two key challenges.
Challenge 1: Which checkpoint to distill from
No single checkpoint universally outperforms the others, and which checkpoint is complementary to the student (i.e., the best validation checkpoint) varies from problem to problem. Statically picking one "strongest teacher" does not work.
Challenge 2: Where to trust the teacher
Even with the right teacher selected, it is only better on local problems. Indiscriminate distillation would overwrite behaviors the student has already mastered, causing capability regression.
GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm proposed by DeepSeekAd, commonly used for LLM reasoning and agent training. Its key feature is that it does not require a separate value network: it samples a group of responses to the same question and normalizes each response's advantage using the mean and standard deviation of rewards within the group, thereby estimating the policy gradient. Compared with PPO, GRPO significantly reduces memory and compute overhead and is more robust to reward scale. During GRPO training, checkpoints are periodically evaluated on the validation set and saved; the "discarded checkpoints" referred to in this article are precisely these intermediate snapshots—they may have already solved certain problems early in training that are later forgotten.
Agentic-OPD's Solution: HCN Selects Teachers, LAD Controls Scope
Agentic-OPD treats the best validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's continuously evolving capability gaps to jointly guide teacher selection and supervision filtering.

HCN Strategy: Dynamically Select Teachers by Unsolved Problem Coverage
The HCN (Highest Coverage Next) strategy dynamically selects teachers based on their weighted coverage of the student's unsolved problems. The set of problems the student currently cannot solve evolves throughout training, and teacher selection adjusts accordingly, avoiding static dependence on a single teacher.
On-policy distillation (OPD) differs from traditional offline knowledge distillation chiefly in the source of student data: offline distillation typically has the student learn on a fixed dataset generated by the teacher, whereas in OPD the student continuously samples trajectories using its own current policy, and the teacher then provides supervision on these "states the student itself would reach." This brings two benefits: first, the training distribution matches the student's actual inference distribution, avoiding exposure bias; second, the teacher only needs to provide guidance in regions where the student genuinely struggles, making supervision more targeted. In Agentic-OPD, the "on-policy" nature of OPD naturally aligns with LAD's local filtering—the student's continuously evolving failure trajectories are precisely the input to distillation.
LAD Module: Restricting Distillation to Locally Superior Regions
The LAD (Locality-Aware Distillation) module restricts knowledge transfer to the student's failure trajectories, and only to problems where the teacher performs better. This way distillation occurs only where the teacher genuinely has a local advantage and does not overwrite capabilities the student has already mastered.
Capability regression (a manifestation of catastrophic forgetting) is common in continual learning and distillation: when a student is forced to imitate a teacher's behavior on some subset, its parameter updates may destroy strategies already mastered on other problems, leading to overall performance degradation. Agentic-OPD uses LAD to restrict distillation to problems where "the student fails and the teacher performs better," essentially a trust-region-style constraint—applying the distillation loss only on samples where the teacher genuinely has a local advantage while keeping the original policy unchanged on all other samples, thereby absorbing new capabilities while protecting existing ones.
Experimental Results: A 4B Student Surpasses Frontier Models
The paper evaluates on five agentic task domains, 24 benchmarks and task subsets. Agentic-OPD improves over its initialization checkpoint by up to 7.4 points. On deep search tasks, a 4B-parameter student model even surpasses the latest frontier models GPT-5.6 and DeepSeek-V4.1.
The significance of this result goes beyond the score. It shows that a completed RL training run can be reinterpreted as a reusable policy pool, and that "post-RL capability recovery" is a stage independent of improvements to the RL algorithm itself.
Implications for Agentic Training Pipelines
The value of Agentic-OPD lies in proposing a new view of resources: checkpoints discarded during training are not garbage but evidence of the regions of capability the model has explored. For teams with limited compute that find it difficult to retrain repeatedly, this "post-hoc mining" approach offers a low-cost path to improving agentic capabilities. It also suggests that the evaluation metric for RL training should not focus only on the final checkpoint—the diversity contained in the training trajectory is itself an asset.
One-line Summary
Uses discarded checkpoints from RL training as a teacher pool to recover lost agentic capability coverage via on-policy distillation.
Paragraph Summary
The article introduces the Agentic-OPD method from an arXiv paper, pointing out that keeping only the best validation checkpoint when training LLM agents with GRPO is lossy: the best checkpoint fails on certain problems that earlier checkpoints from the same training run had already solved, and the union of problems solved across checkpoints far exceeds any single checkpoint. The discarded checkpoints thus form a teacher pool at no additional training cost. The method addresses two challenges—which teacher to pick and where to trust it—using the HCN strategy to dynamically select teachers by weighted coverage of unsolved problems, and the LAD module to restrict distillation to local regions where the student fails and the teacher is better. Experiments on 24 benchmarks across five domains show gains of up to 7.4 points, with a 4B student surpassing GPT-5.6 and DeepSeek-V4.1 on deep search.
Related articles

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Pricing and Quota Breakdown
Google Gemini 4 Neon is now live, benchmark scores closely tracking Carbon, API pricing 3/15 at twice Hydrogen's rate. This article compiles Bilibili creator testing: Neon can replace 90% of Carbon scenarios, AntiGravity quota consumption is only one-third of Carbon, plus subscription channel restrictions and 2API ban risks.

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.