#softmax

Two Major Numerical Bugs in Softmax and Cross-Entropy and Their Fixes

Standard softmax and cross-entropy have two numerical bugs: Softmax Collapse causes non-zero-sum gradients, and Softmax Cancellation produces spurious gradients in distillation. This article presents fixes and their effects on grokking, accuracy stability, and knowledge distillation.