KVQuant
A KV cache quantization method for autoregressive LLMs that introduces sensitivity-weighted non-uniform quantization, outlier handling, and calibration data to reduce accuracy loss at low bit widths.
Timeline (last 90 days)
KVQuant introduces sensitivity-weighted non-uniform quantization, outlier handling, and calibration data to reduce accuracy loss at low bit widths.
The authors adapt KIVI and KVQuant to Fast-dLLM-style prefix caching and dual cache reuse scenarios, and propose two new methods, TRACE and dLLM-KVQ.
KIVI proposes performing per-channel 2-bit quantization on the KV Cache while retaining a small number of outlier channels at full precision, and KVQuant pushes the supported context window to the tens-of-millions-of-tokens scale.
All Facts (3)
KVQuant introduces sensitivity-weighted non-uniform quantization, outlier handling, and calibration data to reduce accuracy loss at low bit widths.
50%UnverifiedThe authors adapt KIVI and KVQuant to Fast-dLLM-style prefix caching and dual cache reuse scenarios, and propose two new methods, TRACE and dLLM-KVQ.
50%UnverifiedKIVI proposes performing per-channel 2-bit quantization on the KV Cache while retaining a small number of outlier channels at full precision, and KVQuant pushes the supported context window to the tens-of-millions-of-tokens scale.
50%