Qwen3-Next
A next-generation large language model in Alibaba's Qwen3 series. It adopts a QSA (Sparse Attention) mechanism to support ultra-long context processing, reducing computational load on long sequences by selectively attending to a subset of tokens.
Timeline (last 90 days)
Qwen3-Next and Kimi Linear both adopt a 3:1 ratio of linear attention to softmax attention.
The project used two RTX 4090 GPUs each with 48 GB of VRAM to run 8 long-context requests each approaching 256K.
The project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
In a coding task test, the model implemented a Python streaming response parser; all 14 self-tests passed on the first attempt, the actual task took approximately 49.94 seconds in total, and an independent re-run of 14 tests plus 4 additional checks all passed.
All Facts (4)
Qwen3-Next and Kimi Linear both adopt a 3:1 ratio of linear attention to softmax attention.
50%UnverifiedThe project used two RTX 4090 GPUs each with 48 GB of VRAM to run 8 long-context requests each approaching 256K.
50%UnverifiedThe project ported a KV offloading approach similar to HiSparse to QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.
50%UnverifiedIn a coding task test, the model implemented a Python streaming response parser; all 14 self-tests passed on the first attempt, the actual task took approximately 49.94 seconds in total, and an independent re-run of 14 tests plus 4 additional checks all passed.
50%