Unverified50% confidenceFactExact time
When the hidden dimension dk is large, the raw attention scores can spike to 20, 30 or even higher, and feeding them directly into softmax leads to output saturation and vanishing gradients.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedThe root cause of attention dilution lies in Softmax normalization: the more tokens in a sequence, the effective attention weight allocated to each position decreases approximately inversely proportional to the sequence length.74% similarUnverifiedStandard attention's computation grows quadratically with sequence length, whereas the low-rank bottleneck decomposes the transformation into two steps: first compressing to r dimensions and then expanding.71% similarUnverifiedAs the number of model layers increases, the parameter share of Attention and FFN rises significantly while the share of Embedding gradually decreases71% similarUnverifiedThe attention score of key evidence has an upper bound, while the maximum score among all effective distractors grows as the number of distractors increases.69% similarUnverifiedWhen resolution doubles, the attention computation the model must handle often grows fourfold or more, causing elapsed time to increase non-linearly.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1010198API
curl https://kongchang.com/api/v1/knowledge/claims/1010198MCP
get_claim(id=1010198)