HiSparse
A KV cache offloading optimization technique for long-context inference in large language models. It offloads cold data to host memory while retaining only hot data in GPU memory, enabling a practical implementation path for ultra-long context inference.
Timeline (last 90 days)
Commit number #60285 in the vLLM GitHub repository was merged into the v0.32.0rc0 tag, with the title "[HiSparse] Report host pool in KV cache capacity logs".
HiSparse is the feature module to which this commit belongs.
v0.30.0 introduces HiSparse, which adds a host tier beneath sparse MLA decoding to form a hierarchical cache structure.
When GPU memory is tight, HiSparse only backfills the missing data that hits the top-k into each request's hot buffer, rather than indiscriminately occupying GPU memory.
The project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
The 12 absorption layers in QSA each independently maintain their own selection strategies, which is incompatible with the assumption that ordinary multi-layer Transformers share a single set of KV indices; therefore, HiSparse's prefetching logic based on cross-layer shared indices cannot be directly reused.
All Facts (6)
HiSparse is the feature module to which this commit belongs.
50%Unverifiedv0.30.0 introduces HiSparse, which adds a host tier beneath sparse MLA decoding to form a hierarchical cache structure.
50%UnverifiedWhen GPU memory is tight, HiSparse only backfills the missing data that hits the top-k into each request's hot buffer, rather than indiscriminately occupying GPU memory.
50%UnverifiedThe project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
50%UnverifiedThe 12 absorption layers in QSA each independently maintain their own selection strategies, which is incompatible with the assumption that ordinary multi-layer Transformers share a single set of KV indices; therefore, HiSparse's prefetching logic based on cross-layer shared indices cannot be directly reused.
50%UnverifiedCommit number #60285 in the vLLM GitHub repository was merged into the v0.32.0rc0 tag, with the title "[HiSparse] Report host pool in KV cache capacity logs".
50%