Unverified50% confidenceDecision RuleExact time
In MoE models, the attention layers, embedding layers and shared layers are used for every token, are small in size, and should be placed in VRAM.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedThe Multi-Query Attention technique allows multiple attention heads to share a single set of Key and Value projection matrices, compressing KV cache memory requirements to 1/H of the original, at the cost of a slight reduction in model expressiveness.73% similarUnverifiedCompressed absorbing attention assigns each layer one of three modes—full, re-indexing, or reuse—to achieve inter-layer KV cache sharing.70% similarUnverifiedDense attention layers require a complete KV Cache, whereas linear layers rely on hidden states.69% similarUnverifiedExisting mainstream sparse attention methods filter interactions by retaining entries with larger values or high-quality regions in the attention matrix.69% similarUnverifiedThe model's layer composition is 54 layers of KDA (Kimi Delta Attention) linear attention, 17 layers of BCSA compressed sparse attention, and 1 full attention layer (layer 72).68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1014110API
curl https://kongchang.com/api/v1/knowledge/claims/1014110MCP
get_claim(id=1014110)