BCSA
A hybrid sparse attention mechanism that retains exact sliding-window attention for recent tokens while compressing distant historical tokens via pooling. A learnable indexing module then retrieves the most relevant compressed blocks, balancing local precision with long-range coverage.
Timeline (last 90 days)
The model's layer composition is 54 layers of KDA (Kimi Delta Attention) linear attention, 17 layers of BCSA compressed sparse attention, and 1 full attention layer (layer 72).
BCSA performs exact windowed attention on the most recent 4,096 tokens, pools more distant context into blocks at a 4:1 ratio, and then uses a learnable indexer to read the top 512 blocks.
All Facts (2)
The model's layer composition is 54 layers of KDA (Kimi Delta Attention) linear attention, 17 layers of BCSA compressed sparse attention, and 1 full attention layer (layer 72).
50%UnverifiedBCSA performs exact windowed attention on the most recent 4,096 tokens, pools more distant context into blocks at a 4:1 ratio, and then uses a learnable indexer to read the top 512 blocks.
50%