log-softmax
The logarithmic form of softmax, computed stably using the log-sum-exp trick to avoid numerical underflow, and commonly used together with cross-entropy.
Timeline (last 90 days)
The fix for Softmax Collapse is to use log-softmax and compute it stably via the log-sum-exp trick.
The fix for Softmax Cancellation is to jointly compute log-softmax and subtract the logit differences directly.
The authors' fixed implementations of log-softmax, softmax, and cross-entropy add no computational complexity.
The fixed implementation keeps gradients effective through stable log-softmax, preventing accuracy collapse.
All Facts (4)
The fix for Softmax Collapse is to use log-softmax and compute it stably via the log-sum-exp trick.
50%UnverifiedThe fix for Softmax Cancellation is to jointly compute log-softmax and subtract the logit differences directly.
50%UnverifiedThe authors' fixed implementations of log-softmax, softmax, and cross-entropy add no computational complexity.
50%UnverifiedThe fixed implementation keeps gradients effective through stable log-softmax, preventing accuracy collapse.
50%