ILRN: An Attention-Free Recurrent Network Using Inverse Lag Memory to Take On Transformer

When Attention Is No Longer the Only Answer

The Transformer architecture has all but dominated the field of sequence modeling, but the computational cost of its self-attention mechanism in long-context scenarios has always been an unavoidable issue. A developer on Reddit, rickey1990, recently shared a project called ILRN (Inverse Lag Recurrent Network), which attempts to replace the attention mechanism with a completely different approach—its core weapon being a mechanism called "Inverse Lag Recurrent Memory" (ILRM).

The motivation behind this project is straightforward: let the recurrent network directly access a weighted record of historical inputs, rather than relying entirely on the hidden state to store information. In other words, the network doesn't need to "remember" everything—it can look back at any time at a historical archive with decaying weights.

How the ILRM Memory Mechanism Is Designed

The core idea of ILRM is not complicated. For every input before the current position, a weight of 1/k is assigned based on its distance k from the current position. This means that closer inputs contribute more, but earlier inputs still retain a direct influence—they are not completely forgotten, but continue to participate in the computation with decreasing weights. The result is a normalized historical representation.

This design is fundamentally different from the hidden state of a traditional RNN. A standard RNN compresses all historical information into a fixed-dimensional vector, and over time the information is severely degraded. ILRM essentially opens a "direct channel" for the network, allowing it to bypass the bottleneck of the hidden state and read the raw history.

At the implementation level, the computation of the inverse lag history is accelerated using causal FFT convolution. However, the update of the recurrent state is still serial—this part cannot be parallelized, and it is also an inherent cost of recurrent architectures.

What Is Causal FFT Convolution?

The convolution theorem states that the convolution of two signals in the time domain is equivalent to their pointwise product in the frequency domain. FFT (Fast Fourier Transform) can reduce the computational complexity of convolution from O(n²) to O(n log n). The so-called "causal" convolution means that the output at position t depends only on t and inputs before it, not on future positions—this is a hard requirement for autoregressive sequence modeling. In ILRM, the weighted sum that assigns a 1/k weight to each historical position is essentially a convolution operation with a fixed kernel function, and can therefore be efficiently implemented using causal FFT convolution, avoiding position-by-position serial accumulation. Similar techniques are also widely used in state space models such as S4 and H3.

Dual Gating and Four-Layer Stacking

The recurrent part of ILRN uses a dual-gating design. One is the access gate, which controls the degree to which the previous recurrent memory contributes to the candidate state; the other is the write gate, which determines how much of the candidate state is updated into the stored memory. This kind of gating structure has similar ideas in GRU and LSTM, but ILRN's gating operates around inverse lag memory.

The actual model, ILRN-4-96, is formed by sequentially stacking four independent ILRM blocks. Each block has its own inverse lag history, two gates, recurrent state, and learnable parameters, with pre/post LayerNorm and residual connections between blocks. The output of the previous layer serves as the input to the next layer—this is a fairly standard way of organizing deep networks.

The Gating Mechanisms of GRU and LSTM

The Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) are two classic variants of recurrent networks, and their core idea is to control the flow of information through learnable "gates." LSTM has an input gate, a forget gate, and an output gate, which respectively determine how much new information is written, how much old memory is retained, and how much of the hidden state is output. GRU simplifies this to two gates, a reset gate and an update gate, with fewer parameters. These gates are usually activated by the sigmoid function, outputting values between 0 and 1 as "switch" coefficients. ILRN's access gate and write gate are functionally similar to these, but what they act on is not the traditional hidden state, but the inverse lag memory store.

Small-Scale Comparative Experiment: ILRN vs Transformer

The author conducted a next-word prediction comparison test on the CNN/DailyMail dataset. The two models had comparable parameter counts, each about 532K parameters, and were each trained on 20 million next-word targets, independently trained under two context windows of 96 and 256.

Top-1 accuracy results on the held-out test set:

Context Length	ILRN	Transformer
96 tokens	27.17%	25.83%
256 tokens	27.42%	24.95%

ILRN achieved higher accuracy under both context lengths. In particular, in the 256-token scenario, Transformer's accuracy dropped from 25.83% to 24.95%, while ILRN actually improved slightly (from 27.17% to 27.42%). This trend suggests that inverse lag memory may have some advantage in longer contexts—but the sample size is too small to draw a conclusion from this.

In terms of speed, ILRN's initial CPU implementation was noticeably slower than Transformer in training efficiency. The author later tried a fused CPU kernel to improve execution speed while keeping the mathematical logic of ILRM unchanged.

Several Issues That Need to Be Viewed Calmly

The author himself clearly stated in the post that these are "preliminary, small-scale experiments with limited training epochs, and cannot serve as evidence that ILRN is generally superior to Transformer." This self-limitation is honest and indeed reflects the limitations of the current experiment.

A model scale of 532K parameters is very small by modern NLP research standards, and 20 million training targets is not sufficient either. The context lengths of 96 and 256 are far from touching the long-sequence scenarios where Transformer truly struggles. More importantly, the experiment was validated on only one dataset, CNN/DailyMail, and lacks evidence of generalization across tasks and domains.

In addition, whether the disadvantage in training speed is fully resolved even after introducing the fused kernel is not given a clear quantitative comparison in the post. The serial nature of recurrent architectures determines that their parallel efficiency on GPUs is difficult to compete with Transformer, and this may become even more prominent after scaling up.

Why Are Recurrent Architectures Difficult to Parallelize?

Transformer's self-attention mechanism can compute all positions in the sequence simultaneously, naturally fitting the large-scale parallel architecture of GPUs. In contrast, the state update of a recurrent network is recursive: computing the state at time t first requires obtaining the state at t-1, and this serial dependency between time steps prevents the many cores of the GPU from being fully utilized. Although convolution-type operations (such as the FFT convolution in ILRM) can be parallelized, the recursion of the recurrent state itself cannot be bypassed. This is also why in recent years SSMs (state space models) such as Mamba have tried to improve parallelism while preserving recurrent characteristics through methods such as selective scanning. As model scale grows, this bottleneck will become increasingly significant.

Open Source and Follow-Up

The author has uploaded the research PDF, architectural formulas, Python implementation, fused kernel, and validation tests to GitHub (repository address: github.com/rickey1990/ILRN). For researchers interested in the revival of recurrent networks and alternatives to attention mechanisms, this project provides a complete reproducible implementation.

The idea of inverse lag memory itself has some novelty—it attempts to introduce an attention-like ability to "directly access history" within the framework of a recurrent architecture, while avoiding the quadratic complexity of attention. But from small-scale experiments to practical usability, there is still a considerable distance to be verified.

One-line Summary

ILRN replaces the attention mechanism with inverse lag memory and surpasses a Transformer of the same parameter count in small-scale experiments.

Paragraph Summary

Reddit developer rickey1990 released ILRN (Inverse Lag Recurrent Network), attempting to replace Transformer's self-attention mechanism with inverse lag memory (ILRM). ILRM allows the recurrent network to directly access historical inputs with 1/k decaying weights, bypassing the information bottleneck of the hidden state, and accelerates computation through causal FFT convolution. The model uses a dual-gated recurrent structure and a stack of four ILRM blocks. On the CNN/DailyMail next-word prediction task, at a scale of 532K parameters, ILRN surpasses a same-scale Transformer at both 96 and 256 context lengths, and at 256 tokens Transformer's accuracy declines while ILRN improves slightly. However, the author emphasizes that the experimental scale is small and the dataset is single, which is not enough to prove general superiority, and training speed remains a weakness of recurrent architectures. The code and paper have been open-sourced on GitHub.