Teacher Model
In knowledge distillation, the model that provides the source of knowledge, whose distribution serves as the learning target for the student model. It is typically obtained through post-training (such as RLHF or rejection-sampling fine-tuning) that yields stronger behavioral preferences.
Timeline (last 90 days)
Reinforcement methods for traditional multi-step diffusion models encounter two structural obstacles on few-step models: the few-step generator is a sampler rather than a score model, and it must be anchored to an undistilled teacher model while the two do not share a transition path.
The study used a 9-billion-parameter open-source model as the executing agent, with a larger teacher model writing notes for it.
Pseudo-labeling refers to using a teacher model to generate predictions for unlabeled or low-confidence samples and using them as temporary labels in training; consistency regularization requires the model to output similar predictions for different augmented views of the same image.
For teams with limited resources, scaling up the teacher model requires allocating a corresponding computational budget to the critic at the same time, otherwise it may backfire.
When the teacher model grows larger while the critic remains small and fixed, the quality of the student model instead degrades, and training collapse even occurs.
Distillation is not simple copying: the student model has limited capacity, so trade-offs must be made in data ratios, temperature parameters, and loss functions, while also being alert to the teacher model's errors being inherited along with everything else.
The limitation of PREEN is that it depends on a fixed 29-parameter editor, and if the editor parameter system changes, the model needs to be re-adapted; in addition, the quality of the pseudo-presets is influenced by the teacher model, and the teacher model's bias may propagate into the final presets.
The training supervision signal of PREEN is generated by introducing a style transfer teacher model: the teacher model's target appearance is inverted into the editor parameter space to generate pseudo-presets.
GRPO passed through only 27% of the teacher's errors, whereas SFT reproduced the teacher's errors almost 100% verbatim.
The teacher model outputs a probability distribution in item space, while the student model makes parallel predictions in code space; the two spaces are inconsistent and cannot be directly aligned.
7 more timeline events
All Facts (17)
Reinforcement methods for traditional multi-step diffusion models encounter two structural obstacles on few-step models: the few-step generator is a sampler rather than a score model, and it must be anchored to an undistilled teacher model while the two do not share a transition path.
50%UnverifiedThe study used a 9-billion-parameter open-source model as the executing agent, with a larger teacher model writing notes for it.
50%UnverifiedPseudo-labeling refers to using a teacher model to generate predictions for unlabeled or low-confidence samples and using them as temporary labels in training; consistency regularization requires the model to output similar predictions for different augmented views of the same image.
50%UnverifiedFor teams with limited resources, scaling up the teacher model requires allocating a corresponding computational budget to the critic at the same time, otherwise it may backfire.
50%UnverifiedWhen the teacher model grows larger while the critic remains small and fixed, the quality of the student model instead degrades, and training collapse even occurs.
50%UnverifiedDistillation is not simple copying: the student model has limited capacity, so trade-offs must be made in data ratios, temperature parameters, and loss functions, while also being alert to the teacher model's errors being inherited along with everything else.
50%UnverifiedThe limitation of PREEN is that it depends on a fixed 29-parameter editor, and if the editor parameter system changes, the model needs to be re-adapted; in addition, the quality of the pseudo-presets is influenced by the teacher model, and the teacher model's bias may propagate into the final presets.
50%UnverifiedThe training supervision signal of PREEN is generated by introducing a style transfer teacher model: the teacher model's target appearance is inverted into the editor parameter space to generate pseudo-presets.
50%UnverifiedGRPO passed through only 27% of the teacher's errors, whereas SFT reproduced the teacher's errors almost 100% verbatim.
50%UnverifiedThe teacher model outputs a probability distribution in item space, while the student model makes parallel predictions in code space; the two spaces are inconsistent and cannot be directly aligned.
50%UnverifiedThe offline design cost of the teacher model and the programs' sensitivity to task distribution are factors that need to be weighed in ABACUS's practical deployment.
50%UnverifiedThe ABACUS team instantiated the framework with Kev: the teacher model plans reasoning steps and generates program code, and the frozen judgment model acts as a semantic executor that scores candidate answers for each step.
50%UnverifiedGenerating training data with a teacher model silently transfers the teacher's blind spots, stylistic habits and failure modes, and the student model inherits the specific shape of the teacher's mistakes.
50%UnverifiedThe original paper proposes that bridging the gap between semi-supervised and fully supervised federated learning hinges on two mutually coupled design axes: the teacher and the anchor.
50%UnverifiedThe teacher and anchor dimensions must work in concert: having only a high-quality teacher without an anchor model may lead to gradual drift, while having only an anchor but poor-quality teacher pseudo-labels makes it difficult to leverage the massive amount of client-side data.
50%UnverifiedCommon directions for improving pseudo-label quality include confidence filtering, introducing language model rescoring (LM rescoring), and ensembling multiple teacher models.
50%UnverifiedOn τ2-bench, as many as 57% of the teacher model's trials are failures.
50%