[KongchangAI]
· 2 min read· 1,412 words

A Visual Guide to Linear Attention: From Delta Rule to Gated DeltaNet and Kimi Linear

A Visual Guide to Linear Attention: From Delta Rule to Gated DeltaNet and Kimi Linear

A Reddit author's visual guide traces linear attention's evolution from early ideas to Gated DeltaNet and Kimi Linear.

This article covers a four-part visual blog by a Reddit author that systematically traces linear attention's technical evolution: from early linear attention to Delta Rule, Gate Rule, Gated DeltaNet, and Kimi Linear. It notes that linear attention reduces complexity from quadratic to linear via recurrent state updates, making it suitable for long-context scenarios; the Delta Rule adds error-correction-style updates and the Gate Rule adds gating selection, together improving state memory efficiency; Gated DeltaNet and Kimi Linear represent this line's move toward engineering deployment. It also notes the original post lacks performance comparison data, making it better as a conceptual entry point, and that the author plans a follow-up infrastructure series.

Why Linear Attention Deserves a Visual Guide

Linear attention isn't a new concept, but over the past few years it has evolved from a "theoretically faster alternative" into a series of engineered models with clear design motivations. A Reddit author has published a four-part visual blog that systematically traces this technical lineage: from early linear attention, to the Delta Rule, the Gate Rule, and on to Gated DeltaNet and Kimi Linear. The author says he read through the key papers and drew illustrations to make both the algorithms themselves and the intuition behind them easier to understand.

The value of this kind of content lies in filling a gap: linear attention papers tend to be formula-dense, while engineering blogs often skip the derivations and jump straight to conclusions. The visual format puts "why it's designed this way" back into a visual context, making it friendlier for readers who want to get started or cross-check their understanding.

reddit source: I wrote a visual guide to Linear Attention — from the original idea to Gated DeltaNet and Kimi Linea

The Core Idea of Linear Attention: Bringing Down Quadratic Complexity

Standard attention requires computing similarity between every pair of positions in a sequence, so complexity grows quadratically with sequence length, and both memory and compute overhead become substantial in long-context scenarios. The basic idea of linear attention is to approximate or restructure the attention computation with kernel functions so that it can be computed recurrently in sequence order, thereby reducing complexity to linear.

This guide starts from "the original idea," which happens to be the prerequisite for understanding later improvements: early linear attention was fast, but had clear shortcomings in expressiveness and memory mechanisms. The subsequent designs—the Delta Rule, the Gate Rule, and others—are essentially patching these shortcomings rather than overturning the linear attention framework.

From Approximation to Recurrence: The Lens of State Updates

Viewing linear attention as a recurrent structure with state is the key to understanding the entire evolutionary line. The model maintains a hidden state at each step and updates it with new input. From this perspective, the differences between variants come down to "how the state is updated"—which is exactly the problem that the Delta Rule and the Gate Rule each address.

Linear attention and recurrent neural networks (RNNs) are highly similar in computational form: both maintain a hidden state that is updated recurrently along the sequence. The difference is that linear attention's state is formed by the outer-product accumulation of key-value pairs, with a fixed state dimension that does not grow with sequence length, so it can be parallelized during training (via prefix sums or chunked scanning), whereas traditional RNNs must execute serially. It is precisely this dual property of "parallelizable in training, recurrent in inference" that gives linear attention an efficiency advantage over standard attention in long-context scenarios. Understanding this helps grasp why subsequent Delta Rule and gating mechanisms focus on state updates.

Delta Rule and Gate Rule: Patching the Memory Update Mechanism

The idea behind the Delta Rule is to make state updates more targeted rather than simply accumulating information. It introduces a mechanism akin to "error correction," so that the model considers the existing state when writing new information, thereby improving memory utilization efficiency. The Gate Rule, meanwhile, approaches from a gating angle, controlling the proportion of information written and retained so that state updates become selective.

These two concepts are prerequisites for understanding Gated DeltaNet. The guide breaks down the Delta Rule and the Gate Rule separately, then introduces their combination—an organization that follows the causal logic of technical evolution: first independent problem awareness, then combined solutions.

Why Gating Matters

In long sequences, indiscriminately accumulating information causes the state to be diluted or overwritten. Gating mechanisms provide a dynamic adjustment tool that lets the model decide which information is worth retaining and which can be discarded. This is in line with the gating ideas from the RNN era, but takes on new implementation forms within the recurrent framework of linear attention.

In linear attention, gating mechanisms are typically implemented per-channel or per-head, i.e., learning a decay coefficient between 0 and 1 for each feature dimension to control how much historical state is retained and how much new information is written. This is intuitively consistent with the forget gate in LSTM, but lighter to implement: no additional nonlinear activation network is needed, and it is usually obtained directly from an input projection. The introduction of gating gives linear attention the ability to "selectively forget," alleviating the problem in ultra-long sequences where a fixed-size state gets filled up by old information and new information cannot be effectively written. Gated DeltaNet precisely combines this gating with the error correction of the Delta Rule to form a more refined state management strategy.

Gated DeltaNet and Kimi Linear: Toward Engineering

Gated DeltaNet can be understood as a class of linear attention architectures that combine the error correction of the Delta Rule with gating mechanisms, aiming to balance memory precision and update efficiency. Kimi Linear is a further exploration along this line, representing an attempt to land linear attention in real model design.

The guide places these two at the end of the evolutionary line, indicating that the author sees them as a comprehensive response to all the earlier design motivations. For readers concerned with long context and inference efficiency, this part is key to understanding "what linear attention can do now."

You may not have noticed that in the Reddit post, the author does not provide detailed performance comparison data for the various models, but instead focuses on algorithmic intuition and visual explanations. This article is therefore better suited as an entry point for conceptual organization than as a basis for model selection. If specific metrics are needed, you still need to go back to the original papers to verify.

Kimi Linear is a linear attention architecture proposed by Moonshot AI, whose core is introducing fine-grained gating and chunked parallel strategies into linear attention to accommodate large-scale training and inference. It is not a single algorithm but a set of design combinations aimed at engineering deployment, involving system-level optimizations such as kernel implementation, chunked scanning, and memory layout. Understanding Kimi Linear therefore requires not only algorithmic intuition but also attention to its parallelization implementation. The infrastructure follow-up series the author plans is precisely aimed at this layer, and is especially relevant to readers working on inference deployment.

How to Use This Guide Effectively

The author's blog is organized into four parts, and it is recommended to read them in order: first establish the basic recurrent perspective of linear attention, then understand the Delta Rule and the Gate Rule in turn, and finally see how Gated DeltaNet and Kimi Linear combine these ideas. The value of the visual format is that you can stop at any time to cross-reference the formulas—you don't have to read it all in one sitting.

The author also mentions plans for a follow-up series exploring linear attention from an AI infrastructure perspective. This means the current guide emphasizes algorithms and intuition, while the infrastructure perspective (such as kernel implementation, memory layout, and parallel strategies) will be supplemented later. Readers working on inference deployment may want to keep an eye on this direction.

The author explicitly welcomes feedback, corrections, and suggestions in the post—an openness that is important for the long-term quality of a technical blog. If you find something in the diagrams that doesn't match the papers, direct feedback is more valuable than quietly skipping over it.

One-line Summary

A Reddit author's visual guide traces the evolution of linear attention from early ideas to Gated DeltaNet and Kimi Linear.

Paragraph Summary

This article introduces a four-part visual blog by a Reddit author that systematically traces the technical evolution of linear attention: from early linear attention, to the Delta Rule and Gate Rule, and on to Gated DeltaNet and Kimi Linear. The article notes that linear attention reduces complexity from quadratic to linear through recurrent state updates, making it suitable for long-context scenarios; the Delta Rule introduces error-correction-style updates and the Gate Rule introduces gating selection, together improving state memory efficiency; Gated DeltaNet and Kimi Linear represent this line's march toward engineering deployment. The article also notes that the original post does not provide specific performance comparison data, making it better suited as a conceptual entry point, and that the author plans a follow-up series from an infrastructure perspective.

#linear attention#Delta Rule#Gated DeltaNet#Kimi Linear#Gate Rule#long context#visual guide
Share:

Related articles