Soft Labels
Probability distribution labels output by a teacher model in knowledge distillation. Compared to hard labels, soft labels carry richer inter-class relational information and are typically smoothed using a temperature parameter during the distillation process.
Timeline (last 90 days)
In soft-label distillation the teacher output carries confidence information, so the student can perceive uncertainty; whereas text-level distillation treats teacher output as deterministic ground truth, so the student inherits the teacher's systematic errors without reservation.
Traditional distillation uses the teacher's output soft labels (i.e., temperature-scaled class probability distributions) as the supervision signal, whereas current mainstream large language model distillation mostly adopts sequence-level supervision in which the teacher generates responses and the student imitates them.
Soft labels are the probability distribution output by the teacher model after softmax normalization, carrying similarity information between classes, and their information content far exceeds that of a single hard label.
Soft labels refer to the probability distribution of outputs rather than simple 0/1 classification results, containing implicit knowledge such as inter-class similarity, and are richer in information than hard labels.
All Facts (4)
In soft-label distillation the teacher output carries confidence information, so the student can perceive uncertainty; whereas text-level distillation treats teacher output as deterministic ground truth, so the student inherits the teacher's systematic errors without reservation.
50%UnverifiedTraditional distillation uses the teacher's output soft labels (i.e., temperature-scaled class probability distributions) as the supervision signal, whereas current mainstream large language model distillation mostly adopts sequence-level supervision in which the teacher generates responses and the student imitates them.
50%UnverifiedSoft labels are the probability distribution output by the teacher model after softmax normalization, carrying similarity information between classes, and their information content far exceeds that of a single hard label.
50%UnverifiedSoft labels refer to the probability distribution of outputs rather than simple 0/1 classification results, containing implicit knowledge such as inter-class similarity, and are richer in information than hard labels.
50%