KDA
A linear attention mechanism variant proposed by Kimi (Moonshot AI). It replaces the progressively growing KV cache with a fixed-size recurrent hidden state, reducing attention computation complexity from O(n²) to O(n) and substantially decreasing memory consumption during long-sequence inference.
Timeline (last 90 days)
根据 vLLM 配置和 Z.ai 文档，GLM-5.3-Flash 的45层模型将 KDA 线性注意力层与 NoPE 稀疏 MLA 层交替堆叠。
The model's layer composition is 54 layers of KDA (Kimi Delta Attention) linear attention, 17 layers of BCSA compressed sparse attention, and 1 full attention layer (layer 72).
KDA is a linear attention mechanism variant that uses a fixed-size recurrent state and does not retain KV cache.
Kimi Linear, introduced by Moonshot AI, is a hybrid linear attention architecture whose core is Kimi Delta Attention (KDA); most layers use KDA, with a full attention block reusing DeepSeek-V3 MLA retained every four layers.
All Facts (4)
根据 vLLM 配置和 Z.ai 文档，GLM-5.3-Flash 的45层模型将 KDA 线性注意力层与 NoPE 稀疏 MLA 层交替堆叠。
50%UnverifiedThe model's layer composition is 54 layers of KDA (Kimi Delta Attention) linear attention, 17 layers of BCSA compressed sparse attention, and 1 full attention layer (layer 72).
50%UnverifiedKDA is a linear attention mechanism variant that uses a fixed-size recurrent state and does not retain KV cache.
50%UnverifiedKimi Linear, introduced by Moonshot AI, is a hybrid linear attention architecture whose core is Kimi Delta Attention (KDA); most layers use KDA, with a full attention block reusing DeepSeek-V3 MLA retained every four layers.
50%