Imprinting
A phenomenon in knowledge distillation whereby a student model, while acquiring task capabilities from the teacher, independently inherits task-irrelevant behavioral tendencies of the teacher, such as safety-versus-pragmatism orientations. The effect is selective and condition-dependent, and cannot be fully eliminated by post-distillation preference optimization.
Timeline (last 90 days)
A study named the phenomenon in which a student model inherits the teacher model's behavioral tendencies on unrelated tasks "Imprinting," with the paper titled "Imprinting: Knowledge Distillation Can Shift Student Behavior."
The strength of Imprinting varies across teacher-student pairs and does not extend uniformly to all forms of safety behavior, indicating that behavioral transfer is selective and condition-dependent.
Excluding a sparse set of "divergence tokens" can significantly reduce the imprinting effect, suggesting that certain specific words or expression patterns in the teacher's responses are the key carriers of behavioral transfer.
After response rewriting, the source-dependence effect still persists and may even introduce additional shifts brought by the rewriter itself, indicating that Imprinting is not driven merely by surface wording.
All Facts (4)
A study named the phenomenon in which a student model inherits the teacher model's behavioral tendencies on unrelated tasks "Imprinting," with the paper titled "Imprinting: Knowledge Distillation Can Shift Student Behavior."
50%UnverifiedThe strength of Imprinting varies across teacher-student pairs and does not extend uniformly to all forms of safety behavior, indicating that behavioral transfer is selective and condition-dependent.
50%UnverifiedExcluding a sparse set of "divergence tokens" can significantly reduce the imprinting effect, suggesting that certain specific words or expression patterns in the teacher's responses are the key carriers of behavioral transfer.
50%UnverifiedAfter response rewriting, the source-dependence effect still persists and may even introduce additional shifts brought by the rewriter itself, indicating that Imprinting is not driven merely by surface wording.
50%