Unverified50% confidenceFactExact time
The attention sink phenomenon was systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the tendency of Transformer models to assign a large proportion of attention weights to the first few tokens at the beginning of a sequence (typically tokens 1–4).
1
Sources
50%
Confidence
Long-term
Relevance
8/31/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedTransformer模型的注意力机制使模型能够在长达数千Token的上下文窗口内追踪对话语义，而早期LSTM模型受限于固定窗口的短期记忆71% similarVerifiedThe self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.71% similarVerifiedThe self-attention mechanism has replaced the previously mainstream RNN/LSTM, and all current mainstream large models are Transformer variants.69% similarUnverifiedLarge models are built on the Transformer architecture at the underlying level; its attention mechanism requires each token to perform interaction computations with all other tokens in the context, resulting in approximately quadratic complexity characteristics68% similarUnverifiedThe attention mechanism of the Transformer architecture allows the model to dynamically attend to information at other positions when processing a sequence, and concentrating requirements discussions within a single continuous conversation produces better results than spreading them across multiple sessions.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/827312API
curl https://kongchang.com/api/v1/knowledge/claims/827312MCP
get_claim(id=827312)