Self-Distillation
A specialized form of model distillation in which the teacher and student are the same model. Knowledge transfer occurs between different versions of the model or across different inference paths of the same model.
Core Facts
Error messages should be written for the model to read and must be actionable, pointing out the actual mistake the model made rather than throwing raw parser errors, which can significantly improve the model's self-correction success rate.
仅用成功轨迹训练的模型往往存在分布外泛化的严重缺陷，遇到工具返回错误时便丧失纠偏能力
Timeline (last 90 days)
RISED introduces a self-distillation mechanism that allows the model to extract supervision signals from its own high-quality trajectories, providing an additional learning source for samples that cannot be learned through group-relative rewards.
Self-Distillation eliminates the need for a separate teacher model, allowing the model to learn from its own historical checkpoints or high-quality outputs generated during current inference.
The UniEvo-VL project proposes enabling vision-language models to achieve self-improvement through self-distillation training, reducing reliance on external supervision signals.
The core difference between self-distillation and traditional knowledge distillation is that the teacher and student can be the same model, or outputs produced by the model itself at different training stages or from different perspectives.
The teacher model in distillation does not have to be larger than the student model; it can be a domain expert model of comparable size.
The three main methods of self-distillation are Self-Distillation Fine-Tuning (SDFT), Self-Distillation Preference Optimization (SDPO), and on-policy self-distillation.
Error messages should be written for the model to read and must be actionable, pointing out the actual mistake the model made rather than throwing raw parser errors, which can significantly improve the model's self-correction success rate.
All Facts (8)
Error messages should be written for the model to read and must be actionable, pointing out the actual mistake the model made rather than throwing raw parser errors, which can significantly improve the model's self-correction success rate.
75%Verified仅用成功轨迹训练的模型往往存在分布外泛化的严重缺陷，遇到工具返回错误时便丧失纠偏能力
70%UnverifiedThe teacher model in distillation does not have to be larger than the student model; it can be a domain expert model of comparable size.
70%UnverifiedSelf-Distillation eliminates the need for a separate teacher model, allowing the model to learn from its own historical checkpoints or high-quality outputs generated during current inference.
50%UnverifiedRISED introduces a self-distillation mechanism that allows the model to extract supervision signals from its own high-quality trajectories, providing an additional learning source for samples that cannot be learned through group-relative rewards.
50%UnverifiedThe core difference between self-distillation and traditional knowledge distillation is that the teacher and student can be the same model, or outputs produced by the model itself at different training stages or from different perspectives.
50%UnverifiedThe UniEvo-VL project proposes enabling vision-language models to achieve self-improvement through self-distillation training, reducing reliance on external supervision signals.
50%UnverifiedThe three main methods of self-distillation are Self-Distillation Fine-Tuning (SDFT), Self-Distillation Preference Optimization (SDPO), and on-policy self-distillation.
50%