Unverified50% confidenceFactExact time
大模型中存在注意力汇聚（Attention Sink）现象，即序列最开头的几个 token 往往保持较高的注意力权重
1
Sources
50%
Confidence
Long-term
Relevance
8/25/2026
First Seen
Sources
Related Entities
Related Claims
Unverified标准Softmax注意力在极长序列上存在注意力湮没（Attention Sink）问题，催生了Sliding Window Attention、RoPE外推、ALiBi等改进方案76% similarUnverified标准Attention的显存消耗随序列长度平方增长，Flash Attention将其压缩到线性增长72% similarUnverified新内核为滑动窗口注意力（SWA）层提供双输入支持，并引入 attention sinks 机制72% similarUnverifiedMulti-head attention performs multiple attention operations in parallel across different low-dimensional subspaces, typically with 8 or 16 heads.71% similarUnverifiedThe computational complexity of standard self-attention is quadratic with respect to sequence length, and achieving a context of one million tokens requires introducing techniques such as sparse attention, sliding window attention, and KV Cache compression.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/798530API
curl https://kongchang.com/api/v1/knowledge/claims/798530MCP
get_claim(id=798530)