KV-Surgeon
A compensation-aware offline KV cache compression method. It uses a quadratic surrogate model to account for the nonlinear coupling among entries caused by softmax renormalization, and jointly performs greedy removal and compensation, advancing the cost-accuracy Pareto frontier on multiple long-context benchmarks.
Timeline (last 90 days)
KV-Surgeon is an offline KV-cache compression method that uses calibration data to account for interactions among cache entries, deriving a quadratic surrogate model to make joint greedy deletion decisions and performing compensation among the remaining entries.
KV-Surgeon was evaluated on three long-context benchmark datasets: QuALITY, LongHealth, and Long Code Arena.
KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.
At every evaluated compression ratio, KV-Surgeon significantly advances the cost-accuracy Pareto frontier.
At a 50x compression ratio, KV-Surgeon improves question-answering accuracy by up to 4 percentage points over the attention-matching method (Zweiger et al., 2026) at comparable cost.
KV-Surgeon presents a faster variant that costs only 40% of the attention-matching method while still surpassing existing state-of-the-art methods.
KV-Surgeon's quadratic surrogate model approximates the effect of a deletion set on the attention output as a quadratic-form function, so that under the greedy framework only a closed-form comparison on the quadratic form is needed at each step to select the single currently optimal deletion, thereby avoiding combinatorial explosion.
KV-Surgeon's compensation term can be computed directly via the closed-form solution of the surrogate model without additional forward passes, so its computational cost is far lower than exhaustive or iterative optimization.
All Facts (8)
KV-Surgeon is an offline KV-cache compression method that uses calibration data to account for interactions among cache entries, deriving a quadratic surrogate model to make joint greedy deletion decisions and performing compensation among the remaining entries.
50%UnverifiedKV-Surgeon was evaluated on three long-context benchmark datasets: QuALITY, LongHealth, and Long Code Arena.
50%UnverifiedKV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.
50%UnverifiedAt every evaluated compression ratio, KV-Surgeon significantly advances the cost-accuracy Pareto frontier.
50%UnverifiedAt a 50x compression ratio, KV-Surgeon improves question-answering accuracy by up to 4 percentage points over the attention-matching method (Zweiger et al., 2026) at comparable cost.
50%UnverifiedKV-Surgeon presents a faster variant that costs only 40% of the attention-matching method while still surpassing existing state-of-the-art methods.
50%UnverifiedKV-Surgeon's quadratic surrogate model approximates the effect of a deletion set on the attention output as a quadratic-form function, so that under the greedy framework only a closed-form comparison on the quadratic form is needed at each step to select the single currently optimal deletion, thereby avoiding combinatorial explosion.
50%UnverifiedKV-Surgeon's compensation term can be computed directly via the closed-form solution of the surrogate model without additional forward passes, so its computational cost is far lower than exhaustive or iterative optimization.
50%