Unverified50% confidenceDecision RuleExact time
跨分词器蒸馏前应先测量熵保留率或概率质量对齐度以避免在错误目标上浪费算力
1
Sources
50%
Confidence
Long-term
Relevance
7/14/2026
First Seen
Sources
跨分词器知识蒸馏：85%信息丢失的根因与链式法则解法
Related Claims
Unverified香农提出的信息熵概念表明低概率事件携带更高的信息量72% similarUnverified不确定性估计主要有两条技术路径：利用Token概率（logprobs）和蒙特卡洛采样（对同一输入进行5-20次高温度采样并计算香农熵）71% similarUnverified从信息论角度看，LLM的输出熵相对较低，倾向于收敛到训练数据的高概率区域70% similarVerifiedCross-entropy loss originates from KL divergence in information theory and measures the degree of difference between the true probability distribution and the predicted probability distribution.69% similarUnverifiedCross-entropy loss often converges faster than MSE in classification tasks because, when paired with softmax output, gradients do not vanish in the saturation region.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/508987API
curl https://kongchang.com/api/v1/knowledge/claims/508987MCP
get_claim(id=508987)