Softmax Dilution

One of three failure mechanisms underlying long-context degradation. When the number of candidate tokens in the context is large, softmax normalization causes attention weights to be distributed more uniformly across candidates, preventing key evidence from receiving sufficiently high attention weights.