Verified75% confidenceFactExact time
The core formula of Self-Attention is Attention(Q,K,V) = softmax(QKᵀ/√dk)V, where dividing by √dk is intended to mitigate the vanishing gradient problem caused by excessively large dot product values.
5
Sources
75%
Confidence
Long-term
Relevance
7/13/2026
First Seen
Sources
Related Claims
UnverifiedLinear attention replaces exp(q^T k) in softmax with a kernel function φ(q)^T φ(k), reducing overall complexity from O(n²d) to O(nd²).76% similarUnverified标准 Scaled Dot-Product Attention 中 Q 与 K 做矩阵乘法得到 n×n 注意力分数矩阵，计算量为 O(n²·d)74% similarVerifiedSelf-attention weights are normalized through the Softmax function, and the sum of the weights is always 1.73% similarUnverifiedThe complete self-attention pipeline is: input matrix → Q/K/V projection → Q×K^T → divide by √dk → Softmax → multiply by V → context-rich output.70% similarUnverifiedQwen3-Next and Kimi Linear both adopt a 3:1 ratio of linear attention to softmax attention.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/500052API
curl https://kongchang.com/api/v1/knowledge/claims/500052MCP
get_claim(id=500052)