KronLSTM: The Kronecker Delta Rule Revolutionizes Long-Context Linear Attention

Linear attention architectures have long faced a trade-off between memory interference and stability in long-context modeling. The recently proposed KronLSTM introduces the Kronecker Delta Rule, providing a minimally invasive update mechanism for associative matrix memory that significantly improves long-context performance while keeping memory dynamics stable. The work has open-sourced its code and models, offering a new analytical perspective on memory design for recurrent linear attention.

The memory dilemma of linear attention and KronLSTM's entry point

Linear attention compresses historical information through a recurrent state, but its associative matrix memory is prone to catastrophic interference during continuous updates—newly written associations can overwrite or distort previously stored content. The DeltaNet family uses the Delta rule for targeted corrections, but each update still produces multi-directional perturbations in matrix space. KronLSTM's core insight is that if each correction proceeds along only a single direction of the matrix, interference can be minimized.

The Kronecker Delta Rule is designed precisely for this. It extends the memory revision mechanism of the xLSTM family, allowing the model to precisely correct target associations while the memory matrix changes along only one direction. This "minimally invasive" property means less destruction of old knowledge, which is especially critical for language modeling tasks that require long-term retention of information.

Linear attention is an efficient alternative to Transformer self-attention: it uses kernel functions to decompose softmax attention into an associatively combinable form, reducing inference-time computational complexity from quadratic in sequence length to linear. Its core is compressing historical information into a recurrent state (usually a key-value association matrix), updating the state each time a new token is read. This "compression-recursion" structure brings constant memory usage and linear time, but it also means all history must be squeezed into a fixed-size memory—once new information conflicts with old information, interference easily occurs. The DeltaNet series was proposed in this context, using the Delta rule (similar to error correction in online learning) to write associations in a targeted way, while KronLSTM further restricts each correction to a single direction to alleviate memory overwriting.

How the Kronecker Delta Rule works

Mathematically, the Kronecker Delta Rule constrains the update to a rank-one structure, i.e., an outer product form. Compared with the multi-rank updates that the standard Delta rule may produce, a rank-one correction naturally has directional singularity. The paper points out that this constraint brings not only reduced interference but also analyzability of memory dynamics.

Specifically, the evolution of the memory matrix can be described as a Markov chain that converges to a stationary distribution. Meanwhile, across the entire input sequence, the expected norm of the memory matrix remains uniformly bounded. This means KronLSTM can maintain stable memory states without additional normalization or stabilization mechanisms—a point that is especially important in long-sequence scenarios, where traditional methods often require careful tuning to prevent state explosion or vanishing.

A rank-one structure means the matrix has rank 1 and can be written as the outer product of two vectors. In linear algebra, a rank-one matrix "stretches" space in only one direction, so when used as an update, the memory matrix changes along only one direction while existing components in other directions remain unaffected. This is the mathematical source of "minimally invasive": compared with the multi-rank perturbations that a general Delta rule may produce, a rank-one correction compresses interference to the smallest possible dimension. The cost is limited expressiveness per update, but KronLSTM accumulates information by repeatedly applying this low-rank correction across the entire sequence, thereby balancing stability and capacity. The term "Kronecker" echoes this outer-product/tensor-product construction.

A Markov chain is a stochastic process in which a state depends only on the current state and is independent of earlier history. Saying that the evolution of the memory matrix "can be described as a Markov chain and converges to a stationary distribution" means that under reasonable input assumptions, regardless of the initial memory, its distribution tends toward a fixed point after enough update steps and does not drift indefinitely. The uniform bound on the expected norm provides a further quantitative guarantee: at any sequence length, the expected size of the memory matrix is capped by the same constant. Together, these two properties explain why KronLSTM does not need an additional normalization layer—stability is built into the update rule itself rather than patched afterward with external tricks.

Experimental performance: competitive on long-context benchmarks

Experiments in the paper show that KronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family. It performs especially well on long-context benchmarks, validating the advantage of minimally invasive memory revision in long-range dependency tasks.

You may not have noticed that KronLSTM achieves these results without introducing complex stabilization components, which reduces the burden of engineering implementation. The open-source code and models have been released on GitHub and Hugging Face, making it easy for the research community to reproduce and extend.

Implications for linear attention design

The value of KronLSTM lies not only in a stronger model but also in providing an analyzable and controllable paradigm for memory updates. By restricting updates to a single direction, memory dynamics gain theoretical guarantees, pointing the way for future research: between expressiveness and stability, perhaps complex trade-offs are not needed—both can be satisfied simultaneously through geometric constraints on the update rule.

For developers of long-context applications, KronLSTM means more reliable long-sequence processing capability without the need to design additional normalization layers for memory states. With the code and models now open, this rule is expected to be integrated into more recurrent architectures, advancing the real-world deployment of linear attention.

One-line Summary

KronLSTM uses the Kronecker Delta Rule to achieve single-direction, minimally invasive memory updates, improving long-context stability.

Paragraph Summary

To address the vulnerability of linear attention's associative matrix memory to catastrophic interference, KronLSTM proposes the Kronecker Delta Rule: constraining each memory correction to a rank-one (outer product) structure so that the memory matrix changes along only a single direction, minimizing damage to old knowledge. The paper proves that under this update, memory evolution can be described as a convergent Markov chain and the expected norm is uniformly bounded, so no additional normalization or stabilization components are needed to maintain stability. Experiments show it is competitive on long-context benchmarks, often surpassing the (Gated) DeltaNet family. The code and models are open-sourced, offering an analyzable new paradigm for memory design in recurrent linear attention.