Unverified50% confidenceFactExact time
SMD also adds a repulsion mechanism: keeping the student away from the teacher's weaker predictions, to avoid inheriting the teacher's unreliable prediction patterns at certain time steps.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedWhen the teacher has non-uniform style preferences, SFT produces off-axis style drift along a non-zero task mean under population updates, which becomes the root cause of forgetting.69% similarUnverifiedValidation loss is computed under teacher-forcing mode, which differs from autoregressive generation at inference time, and therefore cannot expose failure modes such as error accumulation and repetition loops during generation66% similarUnverified教师模型的不确定性在学生token序列上是可因式分解的，分歧被推迟到后续token才解决，可利用链式法则找回65% similarUnverifiedThe key risk of knowledge distillation is that the teacher also makes mistakes, especially on long-tail categories, and its errors are inherited verbatim by the student.65% similarUnverifiedUnder the 4-NFE setting, SMD surpasses existing strong few-step baselines on multiple benchmarks, and even outperforms the multi-step teacher model on some tasks.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1024032API
curl https://kongchang.com/api/v1/knowledge/claims/1024032MCP
get_claim(id=1024032)