KronLSTM
A recurrent model architecture based on linear attention that applies minimally invasive corrections to an associative-matrix memory through the Kronecker Delta rule (a rank-one update), improving long-context modeling capability while maintaining stability of memory dynamics.
Timeline (last 90 days)
KronLSTM provides an update mechanism for associative matrix memory by introducing the Kronecker Delta Rule, improving long-context performance while keeping memory dynamics stable.
The core insight of KronLSTM is that each correction proceeds along only a single direction of the matrix, minimizing interference.
The paper points out that the rank-one constraint of the Kronecker Delta Rule brings reduced interference and analyzability of memory dynamics.
In KronLSTM, the evolution of the memory matrix can be described as a Markov chain and converges to a steady-state distribution.
In KronLSTM, the expected norm of the memory matrix remains uniformly bounded over the entire input sequence, so the memory state stays stable without any additional normalization or stabilization mechanism.
KronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
KronLSTM achieves experimental results without introducing complex stabilization components, reducing the engineering implementation burden.
KronLSTM's open-source code and models have been released on GitHub and Hugging Face.
KronLSTM accumulates information by repeatedly applying low-rank corrections over the entire sequence, thereby balancing stability and capacity.
All Facts (9)
KronLSTM accumulates information by repeatedly applying low-rank corrections over the entire sequence, thereby balancing stability and capacity.
50%UnverifiedThe core insight of KronLSTM is that each correction proceeds along only a single direction of the matrix, minimizing interference.
50%UnverifiedThe paper points out that the rank-one constraint of the Kronecker Delta Rule brings reduced interference and analyzability of memory dynamics.
50%UnverifiedIn KronLSTM, the evolution of the memory matrix can be described as a Markov chain and converges to a steady-state distribution.
50%UnverifiedIn KronLSTM, the expected norm of the memory matrix remains uniformly bounded over the entire input sequence, so the memory state stays stable without any additional normalization or stabilization mechanism.
50%UnverifiedKronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
50%UnverifiedKronLSTM achieves experimental results without introducing complex stabilization components, reducing the engineering implementation burden.
50%UnverifiedKronLSTM provides an update mechanism for associative matrix memory by introducing the Kronecker Delta Rule, improving long-context performance while keeping memory dynamics stable.
50%UnverifiedKronLSTM's open-source code and models have been released on GitHub and Hugging Face.
50%