Unverified50% confidenceFactExact time
标准Softmax注意力在极长序列上存在注意力湮没（Attention Sink）问题，催生了Sliding Window Attention、RoPE外推、ALiBi等改进方案
1
Sources
50%
Confidence
Long-term
Relevance
7/10/2026
First Seen
Sources
GLM-5.2发布：开源编程能力首次逼近闭源旗舰模型
bilibili五里墩茶社6/22/2026
Related Claims
UnverifiedIn the attention mechanism of the Transformer architecture, as sequence length increases, the attention scores of early tokens are diluted by subsequent tokens after softmax normalization, forming an attention sinking effect.78% similarUnverifiedSolutions for streaming processing include causal attention masking, chunked attention, and look-ahead buffering76% similarVerifiedMulti-Head Latent Attention (MLA) is an attention mechanism proposed by DeepSeek-V2.75% similarUnverified标准Attention的显存消耗随序列长度平方增长，Flash Attention将其压缩到线性增长74% similarUnverified新内核为滑动窗口注意力（SWA）层提供双输入支持，并引入 attention sinks 机制74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/453428API
curl https://kongchang.com/api/v1/knowledge/claims/453428MCP
get_claim(id=453428)