Unverified50% confidenceFactExact time
MiniMax-01 and Jamba push the ratio of linear attention to softmax attention to 7:1 or 8:1, and additionally stack an MoE structure on top.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/14/2026
First Seen
Valid until: 1/12/2027
Sources
Related Entities
Related Claims
VerifiedSelf-attention weights are normalized through the Softmax function, and the sum of the weights is always 1.73% similarUnverifiedQwen3-Next and Kimi Linear both adopt a 3:1 ratio of linear attention to softmax attention.70% similarUnverifiedThe computational complexity of standard self-attention is quadratic with respect to sequence length, and achieving a context of one million tokens requires introducing techniques such as sparse attention, sliding window attention, and KV Cache compression.66% similarUnverified标准Softmax注意力在极长序列上存在注意力湮没（Attention Sink）问题，催生了Sliding Window Attention、RoPE外推、ALiBi等改进方案66% similarUnverifiedThe computational cost of standard self-attention grows as O(n²) with sequence length, limiting the scale of context windows64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1029832API
curl https://kongchang.com/api/v1/knowledge/claims/1029832MCP
get_claim(id=1029832)