Paper
A Fix for Softmax and Cross-Entropy
The paper "A Fix for Softmax and Cross-Entropy," written by Barron, Gnanapragasam and Tschannen, points out that standard cross-entropy rounds the gradient of the correct class to zero when the model is highly confident (at a margin of about 5.5 under bf16), causing training instability, and proposes a fix.
Timeline (last 90 days)
Nov 2
The paper "A Fix for Softmax and Cross-Entropy" by Barron, Gnanapragasam and Tschannen (October 2026) points out that standard cross-entropy computation rounds the gradient of the correct answer to exactly zero when the model is sufficiently confident (a margin of about 5.5 under bf16).
Unverified50%