QSA
A sparse attention variant used in the Qwen3-Next model. It performs attention selection at the block level using C4 blocks, where every four adjacent tokens form one computation unit. Twelve dedicated sink layers independently maintain selection strategies to reduce the computational cost of long-sequence inference.
Timeline (last 90 days)
The project added leases, generational tokens, and tail ring buffers, and covered one full 8-way CUDA Graph archive; experiments show that after requests end, KV capacity and cache slots both return to their initial values.
In a same-configuration comparison, single-request throughput at 38K improved from 69.38 tokens per second to 86.07, and 8-way total throughput at approximately 4K improved from 400.16 tokens per second to 464.15.
In the long-context archiving experiment, 8 requests each input approximately 261,000+ tokens and each completed approximately 1,022 outputs; the total throughput of the joint decoding window was 214.85 tokens per second, approximately 26.86 per request, but the window was only approximately 2.28 seconds and prefilling was serial or interleaved.
The author emphasizes that the speeds of individual tasks in multi-agent tasks cannot be added together, and one cannot claim that four lanes are always decoding simultaneously or that there is a fourfold speedup.
The project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
In QSA, a C4 block refers to every 4 adjacent tokens being organized into a single computational unit, with attention selection performed at the block level rather than the token level.
The 12 absorption layers in QSA each independently maintain their own selection strategies, which is incompatible with the assumption that ordinary multi-layer Transformers share a single set of KV indices; therefore, HiSparse's prefetching logic based on cross-layer shared indices cannot be directly reused.
Before the port, each GPU held approximately 1.5 GB of original KV per full-context request, totaling about 12 GB for 8 requests; after the port, 8-way hot cache was approximately 396 MB plus a shared staging area of approximately 1.5 GB, totaling approximately 1.89 GB, a reduction in VRAM requirements of approximately 84%.
Only the cold portion of the KV cache is offloaded; weights, indices, and GDN states remain resident on the GPU, while the full history is stored in host memory and fetched back on demand.
The model introduces QSA, a self-developed absorptive Attention mechanism from Tongyi Qianwen, which is highly similar in concept to DeepSeek's DSA (DeepSeek Sparse Attention).
2 more timeline events
All Facts (12)
Qwen 3.8 Flash Next implements QSA (Qwen Sparse Attention), reducing the overhead of long-context processing through block-level sparse attention.
60%UnverifiedThe project added leases, generational tokens, and tail ring buffers, and covered one full 8-way CUDA Graph archive; experiments show that after requests end, KV capacity and cache slots both return to their initial values.
50%UnverifiedIn the long-context archiving experiment, 8 requests each input approximately 261,000+ tokens and each completed approximately 1,022 outputs; the total throughput of the joint decoding window was 214.85 tokens per second, approximately 26.86 per request, but the window was only approximately 2.28 seconds and prefilling was serial or interleaved.
50%UnverifiedThe author emphasizes that the speeds of individual tasks in multi-agent tasks cannot be added together, and one cannot claim that four lanes are always decoding simultaneously or that there is a fourfold speedup.
50%UnverifiedThe project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
50%UnverifiedIn QSA, a C4 block refers to every 4 adjacent tokens being organized into a single computational unit, with attention selection performed at the block level rather than the token level.
50%UnverifiedThe 12 absorption layers in QSA each independently maintain their own selection strategies, which is incompatible with the assumption that ordinary multi-layer Transformers share a single set of KV indices; therefore, HiSparse's prefetching logic based on cross-layer shared indices cannot be directly reused.
50%UnverifiedBefore the port, each GPU held approximately 1.5 GB of original KV per full-context request, totaling about 12 GB for 8 requests; after the port, 8-way hot cache was approximately 396 MB plus a shared staging area of approximately 1.5 GB, totaling approximately 1.89 GB, a reduction in VRAM requirements of approximately 84%.
50%UnverifiedOnly the cold portion of the KV cache is offloaded; weights, indices, and GDN states remain resident on the GPU, while the full history is stored in host memory and fetched back on demand.
50%UnverifiedIn a same-configuration comparison, single-request throughput at 38K improved from 69.38 tokens per second to 86.07, and 8-way total throughput at approximately 4K improved from 400.16 tokens per second to 464.15.
50%UnverifiedQSA reselects important context at micro-block granularity through a compressed lightweight index, reducing the computational overhead of long-sequence Attention.
50%UnverifiedThe model introduces QSA, a self-developed absorptive Attention mechanism from Tongyi Qianwen, which is highly similar in concept to DeepSeek's DSA (DeepSeek Sparse Attention).
50%