Interaction Effects Between Prompt Breadth and Rollout Refresh in On-Policy Distillation

Background: How Many Prompts Does OPD Actually Need?

On-Policy Distillation (OPD) is an important technical approach in modern large language model training. It has student models generate their own training responses, which are then supervised by a teacher model. A long-standing practical question is: how many distinct prompts does OPD need to achieve optimal performance? And does the answer depend on the student policy used to generate training responses?

A new paper from arXiv (arXiv:2609.25048v1) is the first to systematically study two control variables together — prompt breadth and rollout refresh. These two factors have typically been studied in isolation, but the researchers found a significant interaction effect between them that directly impacts prompt efficiency in OPD.

On-Policy Distillation differs fundamentally from traditional Offline Distillation. In offline distillation, training data is generated once by the teacher model and fixed — the student learns from a static dataset. OPD, by contrast, requires the student model to continuously generate responses (rollouts) using its current policy during training, with the teacher model then scoring or providing probability-based supervision for those responses. This mechanism offers the advantage of "response-policy alignment" — the student learns to correct mistakes it currently makes, rather than imitating a fixed distribution far beyond its current capabilities. However, this also introduces engineering complexity: after each policy update, the distribution of old responses has already shifted, and continuing to train on stale responses introduces "off-policy" bias. The rollout refresh rate is the key hyperparameter controlling this bias — frequent refreshes preserve on-policy properties but significantly increase computational cost.

Experimental Design: A 3×3 Controlled Study Under Fixed Budget

The research team constructed a 3×3 experimental matrix on a mathematical reasoning task. To ensure fair comparison, the study fixed 14,080 trajectories and 110 optimizer updates, varying only two dimensions: the size of the prompt bank and the number of policy snapshots used to generate responses.

The elegance of this design lies in locking down both the compute budget and optimization steps, cleanly separating the contributions of "prompt diversity" and "response freshness," as well as their interaction.

A trajectory here refers to a complete "prompt → response" sample pair — a prompt plus the complete output sequence generated by the student model. Fixing 14,080 trajectories and 110 optimizer updates means the total samples consumed per optimization step remain consistent regardless of prompt bank size. When the prompt bank contains only 8 entries, the same prompts are repeatedly sampled, with the student model generating multiple responses to the same question. When the prompt bank expands to 14,080 entries, each prompt is used on average only once. This design strictly decouples the "source of diversity" at the prompt level from the response level, allowing researchers to independently measure each factor's contribution to final performance without interference from differences in total compute.

Key Finding 1: The Effect of Prompt Breadth Depends on Rollout Refresh

The experimental results revealed a counterintuitive phenomenon — whether increasing prompt breadth helps or hurts depends entirely on whether responses are refreshed.

In the configuration using ten policy snapshots, just 8 prompts achieved an average accuracy of 24.09%, nearly matching the 24.51% achieved with 14,080 distinct prompts. In other words, when responses are sufficiently "fresh," the marginal value of additional prompts drops sharply.

But when responses are frozen to the initial policy, the situation reverses completely: increasing prompt breadth actually dropped accuracy from 21.16% to 19.05%. Under per-update refresh conditions, however, increasing breadth boosted accuracy from 23.61% to 25.57%.

This interaction effect reached 4.07 percentage points, with a 95% problem-pair confidence interval of [2.00, 6.28] — statistically quite robust. This means prompt breadth and rollout refresh are not independently acting factors, but are deeply coupled.

Key Finding 2: A Second Reversal Under Inference Budget

The paper also reveals a more subtle second-layer phenomenon. In matched comparisons across two teacher models, the researchers observed a "second reversal":

• Periodic refresh models perform better under short inference budgets — higher accuracy and better answer completion within short output limits;
• But when output constraints are relaxed to 32K tokens, frozen-response models overtake them in average accuracy, at the cost of consuming 1.7 to 1.8× more response tokens.

This shows that model rankings are not absolute — they shift with inference budget. A solution that wins under constrained compute may not remain ahead when resources are plentiful.

The inference budget, or output token limit, refers to the maximum number of tokens a model may generate when producing an answer. For tasks like mathematical reasoning that require multi-step thinking, output length directly determines how deep the model can develop its reasoning chain. Under a tight token budget, models must provide answers more compactly; under a generous budget (e.g., 32K tokens), models can perform extensive intermediate derivations and even self-verification. Models trained with different strategies tend to excel at different things — "solving problems with short outputs" versus "fully leveraging long outputs." Models trained on frozen responses may learn more verbose reasoning patterns, since the training responses themselves lack diversity and models tend to extend outputs to improve accuracy. Frequently refreshed models, by contrast, have an advantage at more compact reasoning. This finding suggests that single-point benchmark evaluations may systematically favor one class of models, and comprehensive evaluation across multiple inference budget levels is needed.

Practical Implications: Prompt Efficiency Is a Multi-Factor Problem

The value of this research lies in breaking down the oversimplified thinking of "more prompts is always better" or "fewer prompts saves resources." It clearly demonstrates that prompt efficiency in OPD depends simultaneously on rollout refresh strategy and inference budget.

For practitioners, this points to several concrete considerations:

• If the training pipeline can frequently refresh student responses, expanding the prompt bank does yield genuine gains; conversely, blindly increasing prompts under frozen-response conditions is counterproductive.
• Model evaluation should not rely solely on performance at a single output budget — comprehensive assessment across different token budget levels is necessary to avoid one-sided conclusions.
• When designing distillation pipelines, rollout refresh frequency, prompt breadth, and inference budget must be balanced as an integrated whole rather than tuned in isolation.

This is a theoretically and empirically grounded research paper whose findings carry strong practical value for teams optimizing distillation training pipelines — especially in resource-constrained settings where prompt and trajectory budgets must be carefully managed.