Every Token Needs Its Own Teacher: A New Multi-Teacher Adaptive Self-Distillation Method

From "One Teacher for the Whole Run" to "Each Token Picks Its Own Teacher"

In the post-training of large language models (LLMs), On-policy Self-distillation (OPSD) has proven to be an effective technique—it constructs a teacher model conditioned on privileged references to guide the student model in learning from its own generated trajectories. However, existing OPSD methods share a common implicit assumption in their teacher configuration: the teacher is fixed.

According to the paper "Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation," this fixed-teacher approach has fundamental limitations. The authors make a straightforward point: every token needs its own teacher, because even within the same response, teacher preferences can vary from token to token.

The Two Existing OPSD Approaches and Their Shortcomings

The paper categorizes existing approaches into two types. The first keeps the teacher fixed throughout training, always exposing it to the full reference. The second varies the exposure of partial references adaptively within each response, but only across responses—the teacher remains unchanged within a single response.

The authors argue that neither is sufficient. The reason is that different positions within the same response do not rely on reference information to the same degree: some tokens may benefit more from the guidance of the full reference, while others may benefit more from partial references or even different reference combinations. Using a single unified teacher for all tokens effectively erases this fine-grained supervision difference.

Privileged references are a key concept in OPSD: they refer to information available only during training and invisible at inference time, such as human-annotated solution processes, ground-truth answers, or retrieved external documents. The teacher model, conditioned on these references, can produce a higher-quality distribution than the student model itself, while the student model only sees the input question. This information asymmetry is the source of the distillation signal—the teacher "sees more," and the student internalizes this extra information by imitation. Understanding this helps grasp why "how much reference the teacher is exposed to" becomes a tunable design dimension: within a single response, different tokens rely on the reference to different degrees, so the granularity of reference exposure directly determines the precision of the supervision signal.

Teacher-Intervention Pilot Study: Verifying Token-Level Preference Differences

To verify this intuition, the authors designed a teacher-intervention pilot study. The results show that teacher preferences do indeed differ across tokens.

More importantly, the study further reveals that through majority voting and KL-based selection, a "better" teacher can be identified for each token. This provides an actionable technical path for token-level adaptive teacher selection, rather than merely an observation.

KL divergence (Kullback-Leibler divergence) is an asymmetric measure of the difference between two probability distributions. In this context, it is used to compare how much the prediction distribution given by different teacher models at a certain token position deviates from the student distribution (or reference distribution). A smaller KL value indicates that the teacher's prediction at the current position is closer to the target, and thus can be regarded as the "better teacher" for that token. KL-based selection is essentially a per-position distribution-matching score; compared with majority voting, it is more sensitive to the shape of the probability mass distribution, whereas majority voting only cares about whether the highest-probability class is consistent. The two methods provide actionable criteria for token-level teacher selection from different angles.

MTA-OPSD: The First Token-Level Adaptive OPSD Method

Based on these findings, the paper proposes Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), which it claims is the first token-level adaptive OPSD method.

Its core design is to build a multi-teacher pool of candidate teachers and let each token adaptively select its preferred teacher during training. In this way, the granularity of teacher supervision is lowered from "the entire training run" and "a single response" to "a single token," echoing the conclusions of the pilot study.

From a methodological perspective, this idea transforms the teacher-selection problem in self-distillation into a token-level dynamic routing problem: the teacher is no longer a static hyperparameter but a variable that changes with the training state and token context.

On-policy distillation means that the training trajectories are generated by the student model itself rather than from a fixed offline dataset. This aligns the training distribution with the inference distribution and mitigates the exposure bias problem—where the model only sees correct prefixes during training but must face its own accumulated errors at inference time. In OPSD, the teacher provides a supervision signal token by token for each of the student's generated trajectories, and the student updates accordingly. Since trajectories are generated on-policy, the teacher needs to intervene continuously during training, which incurs additional computational overhead and explains why the construction cost of the multi-teacher pool is a practical concern worth attention.

Experimental Results: Consistent Gains on Mathematical Reasoning Tasks

The authors conducted experiments on the Qwen3 model for mathematical reasoning tasks. The results show that MTA-OPSD consistently outperforms existing OPSD methods, with an average gain of up to +5.9 relative to the base model.

The significance of this result goes beyond the number itself. It shows that adaptive adjustment of teacher supervision at the token level can yield measurable performance improvements, thereby supporting the paper's core argument: token-level teacher adaptation is worthwhile. The code is open-sourced in the GitHub repository xingzhejun/MTA-OPSD.

Practical Implications and Open Questions

For engineering teams working on LLM post-training, MTA-OPSD suggests a reusable design principle: when reference information varies in quality or different segments differ in importance, rather than designing an "average-optimal" fixed teacher, it is better to let the supervision signal be selected dynamically at a finer granularity. The combination of a multi-teacher pool and token-level selection also provides a reference paradigm for other forms of distillation and alignment training.

Of course, the paper currently focuses on mathematical reasoning tasks and the Qwen3 model; the construction cost of the multi-teacher pool and the performance of token-level selection on longer contexts and open-domain tasks still require more validation. But the phrase "every token needs its own teacher" has already brought the granularity issue of OPSD to the forefront.

One-line Summary

The paper proposes MTA-OPSD, which lets each token adaptively select a teacher from a multi-teacher pool, achieving gains of up to +5.9 on mathematical reasoning.

Paragraph Summary

The article introduces the paper "Every Token Needs Its Own Teacher," pointing out that existing On-policy Self-distillation (OPSD) methods implicitly assume a fixed teacher, whereas different tokens within the same response do not rely on reference information to the same degree. Through a teacher-intervention pilot study, the authors use majority voting and KL-divergence-based selection to verify that a better teacher can be identified for each token, and then propose MTA-OPSD: building a multi-teacher pool and letting each token adaptively select its preferred teacher during training, lowering the granularity of teacher supervision from the entire training run and a single response down to a single token. Experiments on Qwen3 mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD methods, with an average gain of up to +5.9 relative to the base model, and the code is open-sourced. The article also discusses the practical implications and open questions of the method.