R2S: Fixing Myopic Scorers to Boost Agent Distillation Performance by 12.8%

Title

R2S: Fixing Myopic Scorers to Boost Agent Distillation Performance by 12.8%

Meta Description

This article interprets the R2S method: addressing the myopia problem of teacher token-by-token scoring in agent on-policy distillation, by repairing the scorer without rewards, achieving an average 12.8% success rate improvement across 12 test units, offering a new approach for agent training.

Article Content

In agent training, on-policy distillation (OPD) has attracted much attention because it can leverage teacher model guidance without requiring environment rewards. However, student models often fail at long-horizon tasks due to the "myopia" problem of teacher token-by-token scoring. This article interprets the latest research "Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation," which proposes the R2S method. By repairing the scorer rather than merely adjusting the distillation process, it improves success rates by an average of 12.8% across multiple environments, providing a new approach for agent distillation.

The Myopia Problem of Teacher Scorers

The traditional view attributes OPD failures to compounding errors: after the student deviates from the teacher's reliable region, errors accumulate and lead to task failure. However, researchers found that even starting from the state where the student deviates most, the teacher's own takeover can still complete the task, while the student trained on the teacher's token-by-token scoring often exhausts its turns. This points to a deeper cause—the myopia of next-token scoring.

When the teacher scores each token, it does not know where the turn will end, so safe but delaying actions (such as waiting in place) may receive high scores, while outcome-based reinforcement learning would give zero reward for their eventual failure. This discrepancy between "the teacher's actual behavior" and "the teacher's scoring" is defined by the authors as the rollout-scorer gap.

On-Policy Distillation (OPD) is a knowledge distillation paradigm: the student model generates trajectories on its own policy distribution, and the teacher model scores each token in these trajectories (usually as log probabilities or some scoring), based on which the student optimizes. Unlike offline distillation, OPD's training data comes from the student's current policy, thus alleviating distribution shift; unlike reinforcement learning based on environment rewards, OPD does not require task success/failure reward signals and relies only on the teacher's token-by-token supervision. However, teacher token-by-token scoring is inherently local: each token's score considers only the current context and cannot perceive the long-horizon impact of that action on the final task outcome—this is the root of the "myopia" described in the article.

Compounding error originates from imitation learning theory: the student has a small probability of deviating from the teacher's policy at each step, and once it deviates, subsequent inputs fall outside the teacher's training distribution, causing errors to accumulate and trajectories to collapse. Traditional OPD analyses often attribute failures to this and attempt to expand student coverage through curriculum learning, prefix replay, or teacher intervention. But this paper points out that even if the teacher takes over from the state where the student deviates most, it can still succeed, indicating that the problem is not that the student "cannot walk," but that the teacher's token-by-token scoring cannot distinguish "safe but delaying" from "effectively advancing"—this is a long-horizon blind spot in the scoring signal itself, not merely distribution shift.

R2S: Repair the Scorer First, Then Distill

To address the above problem, the authors propose Rollout-to-Scorer Distillation (R2S). Its core is to repair the teacher scorer before standard OPD, so that its scoring aligns with takeover behavior. The specific process has three steps:

1. Mine High-Divergence States

Starting from an untrained student model, collect the states where it diverges most from the teacher. These states are where the student is most likely to make errors and most needs correct guidance.

2. Teacher Takeover and Reward-Free Filtering

Run teacher takeovers from these states and record complete trajectories. Then, retain only those takeover segments that pass reward-free checks (such as consensus among multiple takeover results or LLM judgment), ensuring the quality of data used for repair. This process requires no task reward signals whatsoever.

Teacher takeover refers to the teacher model continuing to execute subsequent actions from a certain intermediate state (usually a state where the student has deviated significantly) until the end of the turn. Takeover trajectories reflect the teacher's actual behavioral strategy in long-horizon tasks and can serve as a reference for "correct behavior." The key insight of R2S is: teacher takeover itself can complete the task, but teacher token-by-token scoring is inconsistent with takeover behavior; therefore, takeover trajectories are used to calibrate the scorer, rather than directly having the student imitate takeover trajectories.

Reward-free filtering is a key step to ensure the quality of takeover data. Since there are no environment rewards, R2S uses a consensus mechanism (sampling takeovers multiple times and retaining segments with consistent results) or LLM judgment (having a large model judge whether the takeover is reasonable) to filter low-quality trajectories. This avoids fine-tuning the scorer with failed or suboptimal takeovers, while maintaining the reward-free nature of OPD—the entire process does not depend on any task success signal and only utilizes the teacher's own generation capability and external judgment, so it can be generalized to long-horizon interactive tasks where rewards are difficult to define.

3. Fine-Tune the Teacher Scorer

Using the filtered takeover data, fine-tune the teacher model so that it can give scores based on takeover information for each assistant token starting from the divergent turn, and the context does not include the takeover itself when scoring. The repaired teacher is frozen and used as the sole scorer for standard OPD.

Experimental Results: Outperforming Baselines Across the Board

The study compares R2S with three advanced agent OPD methods across 12 test units combining three environments, two teacher models, and four student models. Results show that R2S achieves the best performance in all 12 units, with an average success rate 12.8 percentage points higher than the best baseline in each unit.

This improvement validates the effectiveness of repairing the scorer: by eliminating the rollout-scorer gap, teacher scoring better reflects long-term task outcomes, thereby guiding students to avoid short-sighted behavior.

Significance and Outlook

The proposal of R2S reveals a neglected key problem in agent distillation—the myopia of the scorer. Unlike previous methods that alleviate compounding errors through curriculum learning, prefix replay, or teacher intervention, R2S directly corrects the scoring signal itself, and requires no environment rewards throughout, maintaining the reward-free advantage of OPD.

In the future, this method may be extended to more complex multi-turn interactive tasks and combined with outcome-based reinforcement learning to further enhance agents' long-horizon planning capabilities. For researchers and engineers engaged in agent training, R2S provides a low-cost, high-benefit improvement path.

One-line Summary

R2S fixes the myopia problem of teacher token-by-token scorers, improving agent distillation success rate by 12.8% without rewards.

Paragraph Summary

This article interprets "Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation," pointing out that teacher token-by-token scoring in on-policy distillation (OPD) has myopia: the teacher does not know when the turn will end when scoring each token, causing safe but delaying actions to receive high scores, creating a rollout-scorer gap with the student's actual takeover behavior, rather than the traditionally believed compounding error. The authors propose the R2S method: starting from the states where the untrained student diverges most from the teacher, run teacher takeovers and use reward-free checks to filter high-quality segments, then fine-tune the teacher scorer so that its scoring aligns with takeover behavior, and finally freeze the repaired teacher for standard OPD. Across 12 test units combining three environments, two teachers, and four students, R2S achieves the best performance in all, with an average success rate 12.8 percentage points higher than the best baseline, and requires no environment rewards throughout, maintaining the reward-free advantage of OPD.