Unverified50% confidenceFactExact time
The standard mathematical form of an attention head is softmax(QK^T/√d)V, where Q, K, and V represent the query, key, and value matrices respectively.
1
Sources
50%
Confidence
Long-term
Relevance
11/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSelf-attention maps the input sequence into three sets of vectors — Query Q, Key K, and Value V — computes the dot-product similarity between Q and K, normalizes the result via Softmax to obtain attention weights, and then uses those weights to aggregate the V vectors.73% similarUnverifiedIn QSA, a C4 block refers to every 4 adjacent tokens being organized into a single computational unit, with attention selection performed at the block level rather than the token level.64% similarUnverifiedQKV分别代表Query(查询)、Key(键)和Value(值),命名借鉴自信息检索系统概念63% similarUnverifiedMulti-Query Attention (MQA) lets all query heads share the same set of Key/Value heads, compressing the KV cache to 1/H of its original size (H being the number of heads).62% similarVerifiedGrouped Query Attention (GQA) and Multi-Query Attention (MQA) reduce KV cache memory usage during inference by having multiple query heads share the same group of key/value heads, and are standard optimizations in models such as Llama 3 and Mistral.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1165636API
curl https://kongchang.com/api/v1/knowledge/claims/1165636MCP
get_claim(id=1165636)