[KongchangAI]
ConceptTeacher Model

Teacher Model

In knowledge distillation, the model that provides the source of knowledge, whose distribution serves as the learning target for the student model. It is typically obtained through post-training (such as RLHF or rejection-sampling fine-tuning) that yields stronger behavioral preferences.

Timeline (last 90 days)

Nov 20

Reinforcement methods for traditional multi-step diffusion models encounter two structural obstacles on few-step models: the few-step generator is a sampler rather than a score model, and it must be anchored to an undistilled teacher model while the two do not share a transition path.

Unverified50%
Nov 10

The study used a 9-billion-parameter open-source model as the executing agent, with a larger teacher model writing notes for it.

Unverified50%
Nov 7

Pseudo-labeling refers to using a teacher model to generate predictions for unlabeled or low-confidence samples and using them as temporary labels in training; consistency regularization requires the model to output similar predictions for different augmented views of the same image.

Unverified50%
Nov 2

For teams with limited resources, scaling up the teacher model requires allocating a corresponding computational budget to the critic at the same time, otherwise it may backfire.

Unverified50%
Nov 2

When the teacher model grows larger while the critic remains small and fixed, the quality of the student model instead degrades, and training collapse even occurs.

Unverified50%
Oct 28

Distillation is not simple copying: the student model has limited capacity, so trade-offs must be made in data ratios, temperature parameters, and loss functions, while also being alert to the teacher model's errors being inherited along with everything else.

Unverified50%
Oct 28

The limitation of PREEN is that it depends on a fixed 29-parameter editor, and if the editor parameter system changes, the model needs to be re-adapted; in addition, the quality of the pseudo-presets is influenced by the teacher model, and the teacher model's bias may propagate into the final presets.

Unverified50%
Oct 28

The training supervision signal of PREEN is generated by introducing a style transfer teacher model: the teacher model's target appearance is inverted into the editor parameter space to generate pseudo-presets.

Unverified50%
Oct 26

GRPO passed through only 27% of the teacher's errors, whereas SFT reproduced the teacher's errors almost 100% verbatim.

Unverified50%
Oct 26

The teacher model outputs a probability distribution in item space, while the student model makes parallel predictions in code space; the two spaces are inconsistent and cannot be directly aligned.

Unverified50%

7 more timeline events

All Facts (17)

Unverified

Reinforcement methods for traditional multi-step diffusion models encounter two structural obstacles on few-step models: the few-step generator is a sampler rather than a score model, and it must be anchored to an undistilled teacher model while the two do not share a transition path.

50%
Unverified

The study used a 9-billion-parameter open-source model as the executing agent, with a larger teacher model writing notes for it.

50%
Unverified

Pseudo-labeling refers to using a teacher model to generate predictions for unlabeled or low-confidence samples and using them as temporary labels in training; consistency regularization requires the model to output similar predictions for different augmented views of the same image.

50%
Unverified

For teams with limited resources, scaling up the teacher model requires allocating a corresponding computational budget to the critic at the same time, otherwise it may backfire.

50%
Unverified

When the teacher model grows larger while the critic remains small and fixed, the quality of the student model instead degrades, and training collapse even occurs.

50%
Unverified

Distillation is not simple copying: the student model has limited capacity, so trade-offs must be made in data ratios, temperature parameters, and loss functions, while also being alert to the teacher model's errors being inherited along with everything else.

50%
Unverified

The limitation of PREEN is that it depends on a fixed 29-parameter editor, and if the editor parameter system changes, the model needs to be re-adapted; in addition, the quality of the pseudo-presets is influenced by the teacher model, and the teacher model's bias may propagate into the final presets.

50%
Unverified

The training supervision signal of PREEN is generated by introducing a style transfer teacher model: the teacher model's target appearance is inverted into the editor parameter space to generate pseudo-presets.

50%
Unverified

GRPO passed through only 27% of the teacher's errors, whereas SFT reproduced the teacher's errors almost 100% verbatim.

50%
Unverified

The teacher model outputs a probability distribution in item space, while the student model makes parallel predictions in code space; the two spaces are inconsistent and cannot be directly aligned.

50%
Unverified

The offline design cost of the teacher model and the programs' sensitivity to task distribution are factors that need to be weighed in ABACUS's practical deployment.

50%
Unverified

The ABACUS team instantiated the framework with Kev: the teacher model plans reasoning steps and generates program code, and the frozen judgment model acts as a semantic executor that scores candidate answers for each step.

50%
Unverified

Generating training data with a teacher model silently transfers the teacher's blind spots, stylistic habits and failure modes, and the student model inherits the specific shape of the teacher's mistakes.

50%
Unverified

The original paper proposes that bridging the gap between semi-supervised and fully supervised federated learning hinges on two mutually coupled design axes: the teacher and the anchor.

50%
Unverified

The teacher and anchor dimensions must work in concert: having only a high-quality teacher without an anchor model may lead to gradual drift, while having only an anchor but poor-quality teacher pseudo-labels makes it difficult to leverage the massive amount of client-side data.

50%
Unverified

Common directions for improving pseudo-label quality include confidence filtering, introducing language model rescoring (LM rescoring), and ensembling multiple teacher models.

50%
Unverified

On τ2-bench, as many as 57% of the teacher model's trials are failures.

50%

Source Articles