OPRD: A New Training Method That Frees Coding Agents from "Reminder Dependency"

Long-horizon coding tasks have always been the Achilles' heel of AI agents: a model can write beautiful code snippets in a single-turn conversation, but once it enters a real-world, repository-level development scenario with multi-turn tool calls, it starts dropping constraints and ignoring context. The newly proposed OPRD (On-Policy Reminder Distillation) attempts to solve this problem at the level of the training mechanism.

Diagnosing the Problem: The Answer Is in the Context, but the Agent Can't See It

The paper starts with a diagnostic analysis, and the conclusion is quite sobering: coding agents often fail not because information is missing, but because the relevant information is already in the context and the model simply fails to act on it.

At the same time, the final Pass/Fail signal of SWE-bench-style benchmarks is too sparse—it only tells you whether an entire trajectory succeeded or failed, without indicating which decision step went wrong. This "outcome-oriented" supervision can hardly provide effective gradient signals in long interactions.

The authors categorize two failure modes: first, the model fails to respond correctly to tool feedback; second, the model fails to maintain the constraints established in earlier turns. Both types of errors accumulate as the number of interaction turns increases.

On-Policy vs. Off-Policy Distillation: In knowledge distillation, "policy" refers to the distribution used to generate training data. Off-policy distillation uses trajectories generated by a teacher model as training samples, and the student can only passively imitate the states the teacher has traversed; the problem is that the state distribution visited by the teacher does not match the state distribution the student will actually encounter. Once the student deviates from the teacher's trajectory, there is no corresponding supervision signal—this is distribution shift. On-policy distillation instead requires the student to first generate its own trajectory (i.e., a prefix), and then the teacher provides guidance on the states the student has actually reached. The two state distributions are aligned, and the student receives "correction targeting its current position" at every step, making it generally more stable on long-horizon tasks where errors accumulate easily, at the cost of requiring frequent teacher forward inference calls.

The Core Mechanism of OPRD: Reminders Are Just Training-Time Scaffolding

The design philosophy of OPRD can be summed up in one sentence: provide reminders during training, remove them entirely at inference.

Specifically, it introduces two granularities of "reminder" signals:

• Turn-local Reminder: Affects only the next decision immediately following a given environment observation. It is essentially telling the model, "what that tool just returned means you should do X."
• Global Reminder: Converts the diagnostic results of the final test into trajectory-level guidance, helping the model understand the global intent.

The key point is that these reminders do not enter the inference-stage input. They are temporary scaffolding during training, and the student model must ultimately learn to complete long-horizon tasks independently without reminders.

Outcome-Based Routing: Self-Imitation on Success, Teacher Guidance on Failure

Another design highlight of OPRD is outcome-based routing—using the final result to decide which supervision path each rollout follows:

Successful rollouts follow the autoregressive self-imitation path. The model directly learns from the good trajectories it produced itself, which amounts to self-reinforcement.

Failed rollouts instead receive "reminder-conditioned reverse KL supervision." Specifically, a teacher model re-evaluates the same prefix generated by the student (this time with the Reminder included), and then reverse KL distills the teacher's distributional knowledge back into the student.

The key detail here is on-policy—the student uses its own generated prefix, not data generated by the teacher. This avoids the distribution shift problem common in off-policy distillation.

The teacher and student share the same two-stage initialization: first Coding SFT, then Instruction-Following SFT. This ensures that the teacher's knowledge "connects" to the student's parameter space.

Reverse KL: KL divergence is asymmetric. Forward KL penalizes regions where "the teacher distribution is high and the student distribution is low," tending to make the student cover all of the teacher's modes (mass-covering), which can result in over-smoothing and dispersed generation. Reverse KL penalizes regions where "the student distribution is high and the teacher distribution is low," tending to make the student focus on the teacher's few high-probability modes (mode-seeking). In distillation scenarios, when the teacher has a sharper and more trustworthy distribution because it carries the Reminder, reverse KL can precisely compress this "high-confidence decision-making" into the student network, rather than having the student imitate meaningless low-probability noise in the teacher's tail. This makes it more suitable as a correction target for failed rollouts.

Two-Stage Initialization (Coding SFT + Instruction-Following SFT): The teacher and student share the same initial parameters to ensure "interface compatibility" for distillation. The first stage, Coding SFT, lets the model acquire foundational capabilities such as code syntax, repository structure, and tool-call formats; the second stage, Instruction-Following SFT, aligns instruction understanding and multi-step constraint retention. If the teacher and student come from different initializations, the teacher distribution may fall in regions that the student's parameter space cannot express at all, and reverse KL would pull the student toward an unreachable target, causing distillation to degrade instead. Shared initialization ensures that the teacher's "reminder-conditioned distribution" is merely a directional shift of the student's existing distribution, which the student is capable of fitting.

Benchmarks: An Average Improvement of 11.3 Points

On three mainstream benchmarks, OPRD achieves clear performance gains:

Benchmark	Improvement
SWE-bench Verified	Included in average
SWE-bench Pro	Included in average
SWE-bench Multilingual	Included in average

Relative to the open-source base model Qwen3.5-4B, OPRD improves the average task resolution rate by 11.3 percentage points across the three benchmarks.

Even more convincing is the error-event analysis: under the same diagnostic protocol, the number of aggregated error events dropped from 446 to 222, a reduction of 50.2%. This shows that the improvement is not just "happening to get more problems right," but systematically reducing specific types of decision errors.

Methodological Implications

The value of OPRD may extend beyond coding agents themselves. It proposes a general training paradigm: treat "hints" provided by humans or systems as privileged information during training, rather than as a dependency at inference.

This is the exact opposite of traditional prompt engineering thinking—the latter tries to cram more context into inference to improve performance, while OPRD internalizes contextual assistance into the model parameters. For any agent task requiring long-horizon, multi-turn interaction (not only coding, but also web operation, robotic planning, etc.), this approach has transfer potential.

"Reminder dependency" is essentially a common ailment of current agents: give them sufficiently detailed steps and they perform well, but remove the hints and they show their true colors. OPRD's approach is to acknowledge the existence of this dependency, but gradually desensitize the model during training, ultimately producing a model that can walk without crutches.

One-line Summary

OPRD uses a distillation paradigm of training-time reminders and inference-time removal to improve long-horizon coding agents.

Paragraph Summary

OPRD (On-Policy Reminder Distillation) targets the core pain point of long-horizon coding agents: diagnostics find that agents often fail not because information is missing, but because information already in the context is not adopted, and the final Pass/Fail signal of SWE-bench-style benchmarks is too sparse to locate specific failure steps. OPRD introduces Turn-local and Global Reminders of two granularities as privileged information during training, but removes them entirely at inference, forcing the student model to learn to complete tasks independently. The supervision path is routed by outcome: successful rollouts use self-imitation, while failed rollouts receive teacher reverse-KL distillation with reminders on the student's self-generated prefix, avoiding off-policy distribution shift. On three SWE-bench benchmarks, relative to Qwen3.5-4B, the average task resolution rate improves by 11.3 percentage points, and error events drop from 446 to 222, a reduction of 50.2%.