Concept混合线性注意力机制

Kimi Delta Attention

A hybrid linear attention mechanism used in Kimi K3. By approximating self-attention with kernel functions, it reduces computational complexity from O(N²) to O(N), enabling the model to process million-token long contexts efficiently.

Core Facts

MLA compresses KV pairs into a low-dimensional latent space for caching and recovers the full Keys and Values via an up-projection matrix during inference, compressing the KV cache by several times.

Verified75% confidence3 sources

Source Articles

BeNice! Uses KV Sharing to Save Inference Memory for Looped ArchitecturesOct 23 Transformers v5.20.0 Released: Eight New Models and Export Quantization SupportOct 15 Kimi K3 Released: World's Largest Open-Weight Model with 2.8 Trillion ParametersJul 22 Kimi K3 Deep Dive: Architecture Breakdown and Benchmarks of the 2.8 Trillion Parameter Open-Source LLMJul 22