KIVI
A KV cache quantization method for autoregressive LLMs that quantizes the Key cache per channel and the Value cache per token, exploiting the differing distributional characteristics of K and V.
Timeline (last 90 days)
KIVI quantizes the Key cache per-channel and the Value cache per-token.
The authors adapt KIVI and KVQuant to Fast-dLLM-style prefix caching and dual cache reuse scenarios, and propose two new methods, TRACE and dLLM-KVQ.
KIVI proposes performing per-channel 2-bit quantization on the KV Cache while retaining a small number of outlier channels at full precision, and KVQuant pushes the supported context window to the tens-of-millions-of-tokens scale.
K8V4非对称设计可行的原因是Key参与注意力分数计算对精度敏感，而Value是加权求和对量化噪声有更强鲁棒性，这一发现最早在KIVI等学术论文中被系统验证
All Facts (4)
KIVI quantizes the Key cache per-channel and the Value cache per-token.
50%UnverifiedThe authors adapt KIVI and KVQuant to Fast-dLLM-style prefix caching and dual cache reuse scenarios, and propose two new methods, TRACE and dLLM-KVQ.
50%UnverifiedKIVI proposes performing per-channel 2-bit quantization on the KV Cache while retaining a small number of outlier channels at full precision, and KVQuant pushes the supported context window to the tens-of-millions-of-tokens scale.
50%UnverifiedK8V4非对称设计可行的原因是Key参与注意力分数计算对精度敏感，而Value是加权求和对量化噪声有更强鲁棒性，这一发现最早在KIVI等学术论文中被系统验证
50%