Soft-Label Distillation
A form of knowledge distillation in which the student model learns by minimizing the difference from the teacher model's full probability distribution. In speculative decoding settings, however, it may lead to a decrease in acceptance rate.
Timeline (last 90 days)
Traditional soft-label distillation requires the draft model to approximate the target model's full probability distribution at every position, including high-entropy regions, which is inconsistent with the goal of maximizing acceptance length at inference time.
Soft-label distillation is a form of knowledge distillation in which the teacher model outputs a full probability distribution (soft labels) at each position, and the student model learns by minimizing the difference from the teacher distribution (such as KL divergence).
Soft-label distillation uses a temperature hyperparameter T, dividing logits by T before applying softmax; the larger T is, the smoother the distribution becomes.
All Facts (3)
Traditional soft-label distillation requires the draft model to approximate the target model's full probability distribution at every position, including high-entropy regions, which is inconsistent with the goal of maximizing acceptance length at inference time.
50%UnverifiedSoft-label distillation is a form of knowledge distillation in which the teacher model outputs a full probability distribution (soft labels) at each position, and the student model learns by minimizing the difference from the teacher distribution (such as KL divergence).
50%UnverifiedSoft-label distillation uses a temperature hyperparameter T, dividing logits by T before applying softmax; the larger T is, the smoother the distribution becomes.
50%