Unverified70% confidenceFactExact time
Attention Sinks were systematically described by Xiao et al. from MIT in the 2023 StreamingLLM paper, referring to the phenomenon where Transformer models allocate a large proportion of attention weights to the first 1–4 tokens at the beginning of a sequence.
2
Sources
70%
Confidence
Long-term
Relevance
8/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAttention Sinks 来自麻省理工学院 2023 年的 StreamingLLM 研究75% similarUnverifiedThe paper proposed conceptual frameworks such as the residual stream and attention head composition, using linear algebra language to analyze the attention mechanism inside Transformers.73% similarUnverified大模型中存在注意力汇聚（Attention Sink）现象，即序列最开头的几个 token 往往保持较高的注意力权重67% similarVerifiedIn the attention mechanism of the Transformer architecture, as sequence length increases, the attention scores of early tokens are diluted by subsequent tokens after softmax normalization, forming an attention sinking effect.66% similarUnverifiedTransformer架构的注意力机制决定了模型只能处理当前上下文窗口内的信息66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/823646API
curl https://kongchang.com/api/v1/knowledge/claims/823646MCP
get_claim(id=823646)