MLA
A multi-head latent attention architecture designed by DeepSeek. The DSpark speculative decoding algorithm performs targeted engineering optimizations specifically tailored to the characteristics of this architecture.
Core Facts
DeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
MLA can reduce the KV Cache VRAM usage to 5%–13% of traditional multi-head attention.
Multi-Head Latent Attention (MLA) is an attention mechanism proposed by DeepSeek-V2.
Timeline (last 90 days)
MLA projects K and V into a latent vector space through low-rank joint compression, and at inference time only the compressed latent vectors are cached, compressing the KV cache several times over while maintaining expressive capability.
Models using MLA such as DeepSeek-V4.1-Flash compress the KV state to about 2.4 GB per 32k stream (FP8 KV), while wider architectures consume 3.0 GB–3.8 GB per 32k stream.
The HYV4 architecture integrates four technologies: Multi-head Latent Attention (MLA), DeepSeek Sparse Attention (DSA), gated MLA with learnable attention sinks, and independent Hyperconnections (iHC).
MLA compresses the KV cache per token in GLM from 32,768 numbers down to 576 numbers, a reduction of approximately 98%.
Xingchen 4.0 is built on a data foundation of hundreds of trillions of tokens, adopts architectural designs including MLA, MTP, and MHC, and undergoes staged long-context training from 4K to 32K, 128K, and 256K.
v0.30.0 simplifies the execution flow related to KDA, AttnRes, and MLA, and adds MXFP8 KV cache and asynchronous Engram mechanisms for some models.
The MLA architecture uses a 512-dimensional latent plus 64-dimensional RoPE design, which can reduce KV Cache by approximately one order of magnitude.
DeepSeek-V4's sparse MLA (Multi-head Latent Attention) has achieved end-to-end availability in vLLM v0.28.0, supporting three scenarios: normal decoding, MTP, and DSpark.
MLA integrates low-rank structure into model weights during the training phase, whereas the community experiment applies post-processing approximation to existing models during the inference phase.
The quantized MLX KV cache reduces prompt memory usage by up to 74% compared to before.
3 more timeline events
All Facts (20)
DeepSeek adopts a Mixture of Experts (MoE) architecture and a Multi-head Latent Attention (MLA) mechanism.
80%VerifiedMLA can reduce the KV Cache VRAM usage to 5%–13% of traditional multi-head attention.
75%VerifiedMulti-Head Latent Attention (MLA) is an attention mechanism proposed by DeepSeek-V2.
75%VerifiedGQA (Grouped-Query Attention) and MQA (Multi-Query Attention) were the industry's solutions prior to MLA for alleviating the KV cache problem by sharing Key/Value heads.
75%VerifiedMLA compresses KV pairs into a low-dimensional latent space for caching and recovers the full Keys and Values via an up-projection matrix during inference, compressing the KV cache by several times.
75%UnverifiedMLA中RoPE只对部分维度施加旋转编码，采用分离式设计将一小部分维度专门用于承载RoPE位置编码，与压缩后的内容向量拼接使用
85%UnverifiedDeepSeek V4系列采用了MLA、MoE路由优化以及NSA等创新架构技术
85%UnverifiedThe quantized MLX KV cache reduces prompt memory usage by up to 74% compared to before.
70%UnverifiedDeepSeek's prefix caching implementation is based on the MLA (Multi-head Latent Attention) architecture, which reduces the storage requirements of the KV Cache by tens of times through low-rank compression.
60%UnverifiedMLA projects K and V into a latent vector space through low-rank joint compression, and at inference time only the compressed latent vectors are cached, compressing the KV cache several times over while maintaining expressive capability.
50%UnverifiedModels using MLA such as DeepSeek-V4.1-Flash compress the KV state to about 2.4 GB per 32k stream (FP8 KV), while wider architectures consume 3.0 GB–3.8 GB per 32k stream.
50%UnverifiedThe HYV4 architecture integrates four technologies: Multi-head Latent Attention (MLA), DeepSeek Sparse Attention (DSA), gated MLA with learnable attention sinks, and independent Hyperconnections (iHC).
50%UnverifiedMLA compresses the KV cache per token in GLM from 32,768 numbers down to 576 numbers, a reduction of approximately 98%.
50%UnverifiedXingchen 4.0 is built on a data foundation of hundreds of trillions of tokens, adopts architectural designs including MLA, MTP, and MHC, and undergoes staged long-context training from 4K to 32K, 128K, and 256K.
50%Unverifiedv0.30.0 simplifies the execution flow related to KDA, AttnRes, and MLA, and adds MXFP8 KV cache and asynchronous Engram mechanisms for some models.
50%UnverifiedThe MLA architecture uses a 512-dimensional latent plus 64-dimensional RoPE design, which can reduce KV Cache by approximately one order of magnitude.
50%UnverifiedDeepSeek-V4's sparse MLA (Multi-head Latent Attention) has achieved end-to-end availability in vLLM v0.28.0, supporting three scenarios: normal decoding, MTP, and DSpark.
50%UnverifiedMLA integrates low-rank structure into model weights during the training phase, whereas the community experiment applies post-processing approximation to existing models during the inference phase.
50%UnverifiedTechniques such as GQA and MLA can significantly compress KV cache size; DeepSeek V3 adopts MLA to reduce memory overhead in long-context scenarios.
50%UnverifiedDeepSeek-V2提出MLA（Multi-head Latent Attention），通过低秩压缩将KV Cache压缩至MHA的5%左右
50%