Unverified50% confidenceFactExact time
KL distillation uses the original full-attention model as the teacher and has the student model, with its structure replaced by ZHA, imitate the teacher's next-token distribution, with KL divergence as the loss.
1
Sources
50%
Confidence
Long-term
Relevance
11/2/2026
First Seen
Sources
Related Entities
Related Claims
Unverified知识蒸馏的中间层对齐策略（如PKD、TinyBERT）通过强制学生模型的隐藏表示模仿教师模型的分布间接约束嵌入几何结构70% similarUnverifiedForward KL tends to make the student policy cover all modes of the teacher policy, learning comprehensively but possibly not precisely enough; reverse KL tends to make the student policy concentrate on the high-probability regions of the teacher policy, learning precisely but possibly missing certain modes.68% similarVerifiedThis experiment adopts trajectory-level distillation, which belongs to the category of behavioral cloning, directly having the student model imitate the external behavioral sequences of the teacher model.64% similarUnverifiedThe experiment used knowledge distillation, with Llama 3 8B (token-based) as the teacher model and the byte model as the student model, requiring the design of an alignment mapping between tokens and byte sequences.64% similarUnverifiedDuring training, REMORY uses KL divergence to measure the difference in output distributions between a frozen LLM seeing the full history and seeing the summary plus residual tokens.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1125739API
curl https://kongchang.com/api/v1/knowledge/claims/1125739MCP
get_claim(id=1125739)