Unverified50% confidenceFactExact time
KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedKV cache size is proportional to the number of model layers, the number of attention heads, sequence length, and per-head dimension, and is the primary source of VRAM usage during large model inference.66% similarUnverifiedThe actual memory usage during large model inference is often 1.5–3 times the model weights, due to the overhead from the KV Cache, intermediate activations and context length.65% similarUnverifiedThis solution uses a 4-billion-parameter small model as the draft model, which comes from the same series as the 125-billion main model, achieving a 1.6× to 1.8× speedup.64% similarUnverified模型设计建立在三元量化、稀疏化和细粒度专家混合三项技术的叠加之上63% similarUnverifiedKV-Surgeon is an offline KV-cache compression method that uses calibration data to account for interactions among cache entries, deriving a quadratic surrogate model to make joint greedy deletion decisions and performing compensation among the remaining entries.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1073861API
curl https://kongchang.com/api/v1/knowledge/claims/1073861MCP
get_claim(id=1073861)