Unverified50% confidenceFactExact time
In the teacher-student model, when there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times as fast as the locally connected student (P being the number of input positions).
1
Sources
50%
Confidence
Long-term
Relevance
10/22/2026
First Seen
Sources
Related Entities
Related Claims
Unverified知识蒸馏的中间层对齐策略（如PKD、TinyBERT）通过强制学生模型的隐藏表示模仿教师模型的分布间接约束嵌入几何结构69% similarUnverifiedAnchor divergence differs from metric learning in that it does not learn a fixed linear transformation; instead, it allows users to specify semantic context on demand at inference time through a probabilistic model, without needing to maintain a separate distance matrix or retrain an adaptation layer for each context.65% similarUnverifiedFP4-Local uses quantization-aware distillation (QAD): a BF16 teacher model constrains the student model, the student's routed expert weights are quantized to NVFP4 during the forward pass, and gradient updates go to the underlying BF16 weights.65% similarUnverifiedThe course's A0 assignment requires students to manually implement a linear layer, including explicitly saving the forward input and manually computing backpropagation.63% similarUnverifiedIn knowledge distillation, off-policy distillation uses trajectories generated by the teacher model as training samples, and suffers from the distribution shift problem where the state distribution visited by the teacher is inconsistent with the student's own state distribution.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1083910API
curl https://kongchang.com/api/v1/knowledge/claims/1083910MCP
get_claim(id=1083910)