DeepSeek-V4 Deep Dive: Million-Token Context Architecture Breakthrough and Core Innovations

Introduction: LLMs Enter the Million-Token Era

DeepSeek AI has published the DeepSeek-V4 series paper on arXiv — Towards Efficient Context Intelligence at Million-Token Scale. Authored by over 200 contributors, this work marks a formal transition for large language models into the million-token context era.

Its significance goes far beyond showcasing a 1.6-trillion-parameter MoE giant. The real breakthrough lies in a novel hybrid attention architecture that cuts inference costs for million-token contexts by nearly three-quarters. Tasks that once required 4–10× more compute to handle extremely long texts can now run efficiently on the same hardware.

It's worth noting that the Mixture-of-Experts (MoE) architecture is the foundational infrastructure of the DeepSeek series. Unlike dense models (where every parameter is activated per inference, as in early GPT series), MoE activates only a small subset of "expert" sub-networks per token. DeepSeek-V4 Pro has 1.6 trillion total parameters, but only 49 billion are activated per token — an activation ratio of roughly 3%. This allows the model to hold far more knowledge capacity than a dense model while maintaining inference compute comparable to a much smaller dense model. The MoE routing mechanism determines which experts handle each token; the "Anticipatory Routing" mentioned in V4 is an improvement to this mechanism, designed to predict routing decisions ahead of time during training to reduce instability.

The DeepSeek-V4 series comes in two variants: V4 Pro with 1.6 trillion total parameters and 49 billion activated per token, and the lighter V4 Flash with 284 billion total and 13 billion activated. Both natively support a 1-million-token context window. The most critical figure: at the million-token scale, V4 Pro's per-token inference FLOPs are just 27% of the previous V3.2, and KV Cache size is just 10%.

Why Standard Attention Is the Long-Context Bottleneck

To appreciate V4's value, we must first understand the root of the problem. The Test Time Scaling capability of reasoning models — represented by DeepSeek and OpenAI alike — is fundamentally constrained by the O(N²) computational complexity of attention mechanisms.

Standard Transformer attention has O(N²) complexity, where N is the sequence length (number of tokens). This means doubling the context length quadruples attention compute. At 10K tokens this is manageable, but scaling to 1M tokens means attention compute for a single forward pass is 10,000× that of 10K tokens — practically impossible in real time on current hardware. KV Cache (the mechanism that stores historical token key-value pairs during inference) also balloons to tens or hundreds of gigabytes at the million-token scale, far exceeding single GPU memory limits. This is precisely why Flash Attention, Ring Attention, Sliding Window Attention, and similar techniques have emerged — they all attempt to circumvent this fundamental bottleneck from different angles. At the million-token scale, standard attention is simply not a viable engineering option.

Meanwhile, demand for long-context capability keeps growing: from codebase-level analysis to multi-turn agent tool calls, from cross-document analysis to online learning — all require models to remain coherent at the million-token scale. V3.2 had entered what might be called the "efficiency death zone" at this scale.

V4's approach is clear: attention is fundamentally about maintaining a KV Cache, so rather than compressing model parameters, compress directly at the attention level. The strategy combines two techniques — CSA (lightweight compression + sparse selection) and HCA (aggressive compression + dense attention) — used alternately across different layers, each playing to its strengths.

Core Innovation #1: Hybrid Attention CSA + HCA

CSA: Compressed Absorption Attention

CSA works in three steps. First, every M KV entries are merged into a single compressed entry via Softmax-weighted compression, reducing sequence length by a factor of M. With M=8, a million-token sequence is reduced to just 125K compressed entries.

Next, the Lightning Indexer module selects the most relevant compressed entries — generating Indexer Queries via low-rank projection, computing Index Scores, and using a TopK selector to retain the K highest-scoring entries. CSA's core attention uses Shared KV MQA to reduce memory usage, with grouped output projection to further reduce compute. To preserve local detail, CSA also maintains a sliding window branch that keeps the most recent uncompressed tokens.

The inclusion of sliding window attention has deep academic roots. Early long-context models like Longformer and BigBird demonstrated that for NLP tasks, attention relationships are strongly local — a token's most relevant context is usually within a few hundred tokens. Sliding window attention reduces complexity from O(N²) to O(N·W) (where W is window size) by computing attention only within local windows. CSA thus preserves two complementary information channels: the sliding window captures local semantic coherence (e.g., intra-sentence syntax), while the compression + sparse selection branch handles long-range global retrieval (e.g., recalling key facts from the document's beginning). This design philosophy aligns with aspects of Google Gemini 1.5's architectural exploration.

The elegance of this design: compression ensures efficiency, the sliding window ensures local precision, and sparse selection bridges the two.

HCA: Heavy Compressed Attention

HCA adopts a more aggressive strategy. Its compression ratio M is far larger than CSA's (tens to hundreds of times), merging every M tokens' KV directly into a single entry without any sparse selection, then applying dense attention over the compressed result.

Why does this work? Because the compression ratio is large enough that even ordinary dense attention incurs negligible compute. HCA's architectural philosophy is "simplicity is beauty": when the compression ratio is already extreme, there's no need to add the complexity of sparse selection.

Worthy of note is the FP4 precision acceleration mentioned in the diagram. Numerical precision is a core lever for AI inference efficiency — traditional training used FP32, modern large model training has shifted to BF16, and inference deployment further introduces INT8, INT4, and other quantization formats. FP4 (4-bit floating point) represents the current frontier being explored in academia and industry; each number has only about 15 possible representable values, yet model performance must be maintained under these extreme constraints. NVIDIA's next-generation Blackwell architecture GPUs are making native FP4 support a key optimization priority. DeepSeek-V4's use of FP4 acceleration in Indexer attention computation exemplifies deep integration of cutting-edge quantization techniques with architectural innovation.

In practice, CSA and HCA are interleaved: layers requiring fine-grained long-range selection use CSA, while layers prioritizing maximum efficiency use HCA. The efficiency gains speak for themselves: V4 Pro runs at just 27% of V3.2's inference FLOPs and 10% of its KV Cache, while V4 Flash is even more extreme — FLOPs drop to 10% and KV Cache to just 7%.

Core Innovation #2: MHC Manifold-Constrained Hyper Connections

Standard Hyper Connections expand the residual stream width and decouple residual width from hidden dimensions, but frequently encounter numerical instability when stacking many layers.

MHC's key improvement is constraining the residual mapping matrix to the doubly stochastic matrix manifold (Birkhoff polytope). A doubly stochastic matrix is a non-negative matrix where every row and column sums to 1. These matrices have a critical mathematical property: their spectral norm (largest singular value) is at most 1, meaning multiplying a vector by such a matrix cannot increase its magnitude — i.e., a "non-expansive mapping." One root cause of exploding gradients in deep networks is that certain layer transformation matrices have spectral norms far exceeding 1, causing gradients to be amplified layer by layer during backpropagation.

The matrix is projected onto this manifold using the Sinkhorn-Knopp algorithm with 20 iterations of alternating row-column normalization (alternately normalizing rows to sum to 1, then columns, converging rapidly to a doubly stochastic matrix). This constraint guarantees the matrix's spectral norm stays at most 1, making residual transformations non-expansive and ensuring numerical stability in deep training. Even more elegantly, the set of doubly stochastic matrices is closed under multiplication — no matter how many layers are stacked, the overall mapping's spectral norm never exceeds 1. This approach borrows a core tool from optimal transport theory, a prime example of using mathematical structural constraints to improve deep learning training stability.

Core Innovation #3: MUON Optimizer

V4 replaces AdamW with the MUON optimizer for the majority of parameters, excluding Embeddings and the Prediction Head.

AdamW is currently the most mainstream optimizer for large language model training, maintaining first-moment (momentum) and second-moment (variance) estimates of gradients to adaptively adjust per-parameter learning rates, with weight decay to prevent overfitting. MUON (MomentUm Orthogonalized by Newton-schulz) introduces a critical additional step: before updating parameters, the momentum matrix is orthogonalized via Newton-Schulz iteration, driving its singular values toward 1. The intuition: orthogonalized update directions are more "uniform" across parameter space, avoiding the AdamW problem where some parameter dimensions update too fast while others lag behind. MUON was originally proposed by MIT's Keller Jordan and colleagues, showing superior convergence to AdamW in multiple small-scale experiments; DeepSeek-V4's application at trillion-parameter scale represents the largest industrial validation of this optimizer.

The algorithm flow includes: computing gradients, accumulating Nesterov momentum, orthogonalizing via mixed Newton-Schulz iteration (first eight steps rapidly drive singular values toward 1, last two steps for precise stabilization), followed by RMS rescaling and weight decay. For modules with sparse gradients like Embeddings, AdamW is still used.

At the training infrastructure level, V4 introduces two additional innovations: fine-grained Expert Parallelism that fuses communication and computation into a single kernel to maximize overlap (theoretical 1.92× speedup), and the TileLang DSL for rapid development of fused CUDA kernels.

Experimental Results: Redefining Open-Source LLM SOTA

V4's evaluation covers five dimensions: knowledge reasoning, math, code, long-context, and agent capabilities. Comparison models include Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.6, Kimi K2.6, and GLM 5.1.

Key numbers are striking:

• HLE (Humanity's Last Exam): V4 Pro Max reaches 75.6, the highest among all models, approximately double Gemini 3.1 Pro's corresponding score
• SimpleQA: V4 scores 57.9, leading all open-source models by nearly 20 absolute percentage points
• Million-token long-context MRCR task: V4 reaches 83.5, surpassing Gemini 3.1 Pro's 76.3
• Code capability: V4 Pro achieves 3052 ELO on Codeforces; ties Gemini 3.1 Pro at 80.6 on SWE-bench Verified

Notable is V4 Flash — as a lightweight model with only 13 billion activated parameters, its reasoning performance approaches the previous-generation open-source SOTA, making it an exceptional value proposition. The consistent improvement from Non-Think to High to Max mode also validates that investing more Test Time Compute on V4's architecture genuinely translates to better performance.

Test Time Scaling is one of the most important paradigm shifts in AI over the past two years: rather than piling on more training data and parameters, invest more compute at inference time to let the model "think longer." The success of OpenAI's o1/o3 series and DeepSeek-R1 validated this approach — through Chain-of-Thought, Tree of Thought, Best-of-N sampling, and related techniques, models can significantly improve complex reasoning by extending their inference-time thinking chains. V4's Non-Think, High, and Max modes are a direct embodiment of this philosophy. The million-token context window is especially critical for Test Time Scaling, since complex multi-step reasoning chains themselves consume enormous numbers of tokens — long context is the infrastructure underpinning deep reasoning.

Limitations and Future Outlook

Despite impressive results, V4 has clear limitations.

Architectural: To reduce risk, the team retained many components already validated in V3, resulting in a relatively complex overall architecture. The root causes of training instability remain unclear — while measures like Anticipatory Routing mitigate the issue in engineering terms, the underlying mechanisms are not yet fully understood.

Capability: On pure knowledge evaluations (e.g., MMLU Pro and SimpleQA), V4 still trails Gemini 3.1 Pro. Reasoning capability lags frontier closed-source models by roughly 3–6 months. Agent capability scores ~67% on internal coding benchmarks versus Opus 4.5's 80%, and about 9% of internal users report low-level errors and over-thinking behavior.

V4's two-stage post-training pipeline is worth highlighting: starting from the Base model, domain experts are independently cultivated (SFT fine-tuning + GRPO reinforcement learning), then On-Policy Distillation is used to distill over 10 teacher models into a single student model — avoiding the performance degradation typical of traditional weight merging.

The authors' future directions include: architectural simplification and distillation, exploring new dimensions such as sparse embeddings, low-latency long-context deployment, and the much-anticipated multimodal capabilities.

Conclusion

DeepSeek-V4's core lesson is: through architectural innovation rather than simply stacking parameters, million-token contexts can move from theoretical possibility to practical engineering reality. The CSA + HCA hybrid attention makes million-token context a standard feature rather than a luxury; MHC + MUON addresses deep training stability; and FP4 quantization, mixed-precision KV Cache, and fused kernels provide deep optimization across quantization, memory, and compute dimensions. As long-context and agent intelligence become increasingly critical, V4 delivers an engineering answer well worth studying.

Key Takeaways