Privileged Context
Additional information visible only to the teacher in self-distillation (such as source material, answers or domain documents), used to compute the teacher's next-token distribution, which is compared with the student model's unconditional distribution to determine supervision signals.
Core Facts
Timeline (last 90 days)
LEAF-OPD gives the teacher model an additional instruction (privileged context) on forget data to guide the student to learn to refuse, while on retain data the teacher does not use privileged context; the student always runs without privileged context.
Privileged context in self-distillation refers to additional information visible only to the teacher, such as source material, answers, or domain documents.
Squint OPSD adopts a teacher-student distillation paradigm, where the teacher model receives a downsampled image view and is given the ground-truth text transcription as privileged context.
The concept of privileged context originates from the Learning Using Privileged Information (LUPI) framework, which provides additional information during training that is unavailable at inference time to improve learning.
All Facts (4)
The concept of privileged context originates from the Learning Using Privileged Information (LUPI) framework, which provides additional information during training that is unavailable at inference time to improve learning.
75%UnverifiedLEAF-OPD gives the teacher model an additional instruction (privileged context) on forget data to guide the student to learn to refuse, while on retain data the teacher does not use privileged context; the student always runs without privileged context.
50%UnverifiedPrivileged context in self-distillation refers to additional information visible only to the teacher, such as source material, answers, or domain documents.
50%UnverifiedSquint OPSD adopts a teacher-student distillation paradigm, where the teacher model receives a downsampled image view and is given the ground-truth text transcription as privileged context.
50%