MTA-OPSD
The first token-level adaptive on-policy self-distillation method. It constructs a pool of multiple teachers, allowing each token to adaptively select a preferred teacher during training, thereby lowering the granularity of teacher supervision to the level of individual tokens.
Timeline (last 90 days)
The core design of MTA-OPSD is to construct a multi-teacher pool of candidate teachers and let each token adaptively select its preferred teacher during training.
Experiments on Qwen3 models for mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD methods, with an average gain of up to +5.9 relative to the base model.
The code for MTA-OPSD has been open-sourced in the GitHub repository xingzhejun/MTA-OPSD.
The paper currently focuses on mathematical reasoning tasks and Qwen3 models; the construction cost of the multi-teacher pool and the performance of token-level selection on longer contexts and open-domain tasks still require more validation.
MTA-OPSD transforms the teacher selection problem in self-distillation into a token-level dynamic routing problem, where the teacher is no longer a static hyperparameter but a variable that changes with training state and token context.
All Facts (5)
The core design of MTA-OPSD is to construct a multi-teacher pool of candidate teachers and let each token adaptively select its preferred teacher during training.
50%UnverifiedExperiments on Qwen3 models for mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD methods, with an average gain of up to +5.9 relative to the base model.
50%UnverifiedThe paper currently focuses on mathematical reasoning tasks and Qwen3 models; the construction cost of the multi-teacher pool and the performance of token-level selection on longer contexts and open-domain tasks still require more validation.
50%UnverifiedMTA-OPSD transforms the teacher selection problem in self-distillation into a token-level dynamic routing problem, where the teacher is no longer a static hyperparameter but a variable that changes with training state and token context.
50%UnverifiedThe code for MTA-OPSD has been open-sourced in the GitHub repository xingzhejun/MTA-OPSD.
50%