SnapKV
A KV cache eviction and compression method based on attention scores that selectively removes cache entries for tokens deemed less important. It is designed to reduce GPU memory usage during large language model inference without retraining.
Timeline (last 90 days)
ChunkKV, proposed by the Hong Kong University of Science and Technology (Guangzhou) team, uses continuous semantic chunks as the basic compression unit, retaining or discarding entire chunks. In the NIAH benchmark test, with the KV cache limited to 128, the LLaMA-3-based ChunkKV achieves an accuracy of 73.8%, exceeding SnapKV's 58.9%.
Score-based eviction methods such as SnapKV and H2O use attention scores to determine which tokens are unimportant and delete them.
H2O is a dynamic sparse method based on Top-k selection, SnapKV is a method based on cumulative attention scores, and MInference is a structured sparse method with block sparsity.
All Facts (3)
ChunkKV, proposed by the Hong Kong University of Science and Technology (Guangzhou) team, uses continuous semantic chunks as the basic compression unit, retaining or discarding entire chunks. In the NIAH benchmark test, with the KV cache limited to 128, the LLaMA-3-based ChunkKV achieves an accuracy of 73.8%, exceeding SnapKV's 58.9%.
50%UnverifiedScore-based eviction methods such as SnapKV and H2O use attention scores to determine which tokens are unimportant and delete them.
50%UnverifiedH2O is a dynamic sparse method based on Top-k selection, SnapKV is a method based on cumulative attention scores, and MInference is a structured sparse method with block sparsity.
50%