Unverified50% confidenceFactExact time
Without weight decay, unbounded logit growth causes numerical underflow in standard softmax, making gradients vanish entirely and manifesting as a sudden accuracy collapse.
1
Sources
50%
Confidence
Long-term
Relevance
10/23/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSoftmax Collapse causes the gradient of the target logit to be rounded to zero, breaking the property that gradients sum to zero.76% similarUnverifiedLow-bit Adam collapses in rare-input scenarios because the gradient signal produced by rare samples is weak, and after coarse-graining the momentum the weak signal is overwhelmed, causing the corresponding parameters to fail to update.71% similarUnverifiedThe gradient imbalance caused by Softmax Collapse may lock the model prematurely into a memorization solution, delaying or preventing the transition to a generalization solution.70% similarUnverifiedAs long as the radius is no longer a fixed constant, the radial fragility based on the mean-curvature interpretation quickly fails.69% similarUnverifiedSoftmax函数具有平移不变性，任意放大所有logit的尺度不改变预测类别，却会将输出概率压向极端值，是其天然的过度自信倾向69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1088274API
curl https://kongchang.com/api/v1/knowledge/claims/1088274MCP
get_claim(id=1088274)