Extreme-Value Interference
A phenomenon in the attention mechanism whereby, as the number of effective distractors increases, the maximum attention score among distractors grows at a rate of √log N and approaches the score of genuine evidence, preventing the model from effectively distinguishing key evidence from distractor content.
Timeline (last 90 days)
The paper abstracts the context degradation phenomenon as a problem of extreme-value interference in attention.
The paper derives that under the softmax retrieval framework, the evidence score margin required to maintain a fixed accuracy must grow at a rate of Ω(√log N), where N is the number of effective distractors rather than the context length.
All Facts (2)
The paper abstracts the context degradation phenomenon as a problem of extreme-value interference in attention.
50%UnverifiedThe paper derives that under the softmax retrieval framework, the evidence score margin required to maintain a fixed accuracy must grow at a rate of Ω(√log N), where N is the number of effective distractors rather than the context length.
50%