Unverified50% confidenceFactExact time
The complete self-attention pipeline is: input matrix → Q/K/V projection → Q×K^T → divide by √dk → Softmax → multiply by V → context-rich output.
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedThe core formula of Self-Attention is Attention(Q,K,V) = softmax(QKᵀ/√dk)V, where dividing by √dk is intended to mitigate the vanishing gradient problem caused by excessively large dot product values.70% similarUnverified标准 Scaled Dot-Product Attention 中 Q 与 K 做矩阵乘法得到 n×n 注意力分数矩阵，计算量为 O(n²·d)67% similarVerifiedFor a model with L layers, H attention heads, and per-head dimension d, caching n tokens requires VRAM of 2 × L × H × d × n × (precision bytes).64% similarUnverifiedIQuest-Q1 employs a mixed ratio of 3 layers of sliding window attention combined with 1 layer of full attention, with only 25 out of 88 layers expanding the KV cache as context grows.63% similarUnverifiedThe paper proposed conceptual frameworks such as the residual stream and attention head composition, using linear algebra language to analyze the attention mechanism inside Transformers.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1010201API
curl https://kongchang.com/api/v1/knowledge/claims/1010201MCP
get_claim(id=1010201)