Unverified50% confidenceFactExact time
The paper "A Fix for Softmax and Cross-Entropy" by Barron, Gnanapragasam and Tschannen (October 2026) points out that standard cross-entropy computation rounds the gradient of the correct answer to exactly zero when the model is sufficiently confident (a margin of about 5.5 under bf16).
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/2/2026
First Seen
Valid until: 1/31/2027
Sources
Related Entities
Related Claims
UnverifiedFinite-sample estimation of entropy rate faces a systematic negative bias problem, where plug-in estimators tend to underestimate the true entropy; this can be compensated using the Miller-Madow correction by adding a (k-1)/(2N) term.72% similarUnverifiedUnder the uniform transition case, the entropy rate of a Markov chain reaches its maximum value of log N, while under the deterministic transition case the entropy rate is 0.71% similarUnverifiedAssuming the data follows a Bernoulli or multinomial distribution, maximizing the likelihood is equivalent to minimizing the cross-entropy loss function.69% similarVerifiedThe cross-entropy loss in classification tasks is essentially the negative of the log-likelihood function, and minimizing cross-entropy is equivalent to maximizing the log-likelihood of the model on the training data.68% similarUnverified交叉熵损失函数 H(p,q) = -Σp(x)log q(x) 的最小值在q=p时取到，等于真实分布的熵67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1127255API
curl https://kongchang.com/api/v1/knowledge/claims/1127255MCP
get_claim(id=1127255)