R2S
A teacher-scorer repair method for online policy distillation of agents. By mining high-divergence states, filtering through teacher takeover and no-reward selection, and fine-tuning the teacher scorer, it aligns token-level scores with long-horizon behavior without requiring environment reward signals.
Timeline (last 90 days)
The core of the R2S (Rollout-to-Scorer Distillation) method is to repair the teacher scorer before standard OPD so that its scores align with the teacher's takeover behavior.
The R2S pipeline has three steps: 1) starting from the untrained student model, mine the states where it diverges most from the teacher; 2) run teacher takeovers from these states and filter the takeover segments using reward-free checks (consensus or LLM judging); 3) fine-tune the teacher scorer on the filtered takeover data, then freeze it as the sole scorer for standard OPD.
Across 12 test cells spanning three environments, two teacher models, and four student model combinations, R2S achieved the best performance in all 12 cells, with an average success rate 12.8 percentage points higher than the best baseline in each cell.
In the experimental comparison, R2S was compared against three state-of-the-art agentic OPD methods and comprehensively outperformed these baselines.
Unlike previous methods that mitigate compounding errors through curriculum learning, prefix replay, or teacher intervention, R2S directly corrects the scoring signal itself and requires no environment rewards throughout.
All Facts (5)
The core of the R2S (Rollout-to-Scorer Distillation) method is to repair the teacher scorer before standard OPD so that its scores align with the teacher's takeover behavior.
50%UnverifiedThe R2S pipeline has three steps: 1) starting from the untrained student model, mine the states where it diverges most from the teacher; 2) run teacher takeovers from these states and filter the takeover segments using reward-free checks (consensus or LLM judging); 3) fine-tune the teacher scorer on the filtered takeover data, then freeze it as the sole scorer for standard OPD.
50%UnverifiedAcross 12 test cells spanning three environments, two teacher models, and four student model combinations, R2S achieved the best performance in all 12 cells, with an average success rate 12.8 percentage points higher than the best baseline in each cell.
50%UnverifiedIn the experimental comparison, R2S was compared against three state-of-the-art agentic OPD methods and comprehensively outperformed these baselines.
50%UnverifiedUnlike previous methods that mitigate compounding errors through curriculum learning, prefix replay, or teacher intervention, R2S directly corrects the scoring signal itself and requires no environment rewards throughout.
50%