Unverified50% confidenceFactExact time
The purpose of the scaling factor √d is to prevent the dot product results from becoming too large as the dimension increases, which would cause the softmax output to approach one-hot and the gradients to vanish.
1
Sources
50%
Confidence
Long-term
Relevance
11/2/2026
First Seen
Sources
Related Entities
Related Claims
Unverified计算 softmax 函数时指数运算可能导致浮点数溢出，标准做法是先减去输入向量的最大值再取指数76% similarUnverifiedThe softmax operation is prone to numerical overflow or underflow due to insufficient precision when handling extreme values, which is the core technical reason why early FP8 quantization performed poorly.72% similarUnverifiedThe gradient imbalance caused by Softmax Collapse may lock the model prematurely into a memorization solution, delaying or preventing the transition to a generalization solution.71% similarUnverified缩放点积将点积除以√d（d为嵌入维度），使点积方差保持在合理范围，让Softmax输出更平滑的概率分布以稳定训练69% similarUnverifiedWithout weight decay, unbounded logit growth causes numerical underflow in standard softmax, making gradients vanish entirely and manifesting as a sudden accuracy collapse.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1127268API
curl https://kongchang.com/api/v1/knowledge/claims/1127268MCP
get_claim(id=1127268)