PLUG and ILRM: Replacing Attention with Harmonic Series to Solve RNN Long-Sequence Modeling

The Classic RNN Problem and a New Solution

In an era dominated by Transformers and attention mechanisms, recurrent neural networks (RNNs) seem to have taken a back seat. Yet the inherent vanishing gradient problem — the model's difficulty in capturing long-range dependencies — remains a classic challenge that has never been fully resolved.

Recently, a developer published an open-source project called novel-rnn-architectures on Reddit and GitHub, proposing two new RNN architectures: PLUG and ILRM. Both architectures share a clear goal — solving the vanishing gradient problem in long-sequence modeling without relying on attention modules.

This approach stands out in today's landscape. Rather than stacking attention layers on top of RNNs, it attempts to fundamentally change how gradients propagate by rethinking the mathematical mechanisms inside the recurrent structure itself.

Core Mechanism: How Harmonic Series Reshapes Memory Decay

From Exponential to Polynomial Decay

The core pain point of traditional RNNs (including GRU and LSTM) is that historical information decays at an exponential rate during backpropagation. The longer the sequence, the closer to zero the gradients of early tokens become, making it increasingly hard for the model to learn long-range dependencies.

The key innovation proposed by this project is feeding each recurrent unit a deterministic, normalised inverse-lag history, denoted $P_t$, which encodes a weighted summary of all previous inputs.

The crucial detail lies in the weighting scheme — weights are determined by harmonic numbers, so historical information decays at a rate of $1/k$ rather than exponentially. The authors further show that the direct gradient path from the current token back to historical inputs decays at the polynomial rate of $\frac{1}{t\ln t}$, which is far slower than exponential decay.

A Stable Gradient Horizon

This design directly creates a stable gradient horizon. Even as sequence length grows to thousands of steps, early inputs can still maintain meaningful gradient signals and continue to participate in the learning process.

From an engineering perspective, this is an elegant design tradeoff: rather than letting every position "see" all other positions like attention does (at the cost of $O(T^2)$ complexity), a carefully designed mathematical decay function gives the recurrent structure itself the ability to maintain long-term memory.

Computational Efficiency: FFT Acceleration Brings Complexity Down to O(T log T)

Computational efficiency is another major bottleneck for long-sequence modeling. If calculating historical information requires traversing the entire sequence naively, the cost would be prohibitive.

The project applies a classic but effective trick: computing the history vector $P_t$ efficiently using zero-padded FFT linear convolutions, keeping overall complexity at $O(T \log T)$.

This is more efficient than the $O(T^2)$ of standard attention, aligning with the optimization targets of many recent efficient Transformer variants. Converting convolution operations to the frequency domain for acceleration is a well-established technique in signal processing — applied here to optimize RNN history computation, it reflects a nice cross-domain synthesis.

Experimental Results: Extrapolating from 64-Step Training to 4096 Steps

The author is candid about the experimental limitations: due to local hardware constraints, tests were only run at small scale on CPU using synthetic benchmarks. Even so, the preliminary results show notable promise.

Long-Range Extrapolation Capability

The most striking result comes from the long-range extrapolation task:

• The model was only trained on delays of 16–64 steps
• Both PLUG and ILRM successfully extrapolated to 4,096 steps, achieving 100% accuracy on a one-bit information retention task
• A standard GRU baseline degraded to random-guessing level

This ability to "train on short sequences, test on much longer sequences" is precisely the kind of generalization that long-sequence modeling demands most.

Gradient Stability Comparison

The comparison in gradient propagation is equally striking:

• On sequences of 1,024 steps, PLUG and ILRM maintained average gradients at the initial token stably around ~10⁻⁴
• Standard GRU exhibited absolute numerical underflow, with gradients as low as ~10⁻¹⁴⁵ — essentially zero

This comparison directly validates the theoretical design: polynomial decay genuinely sustains effective long-distance gradient signals far better than exponential decay.

Honest Assessment: Promise and Open Questions

What Deserves Credit

The project provides a complete mathematical framework in PDF form and a runnable PyTorch implementation, with minimal code versions for both models. This "theory + code" open-source approach means anyone can reproduce, verify, or critique the mathematical derivations.

The author is also notably transparent, clearly labeling the experiments as preliminary and small-scale, and actively inviting community feedback on both the math and the implementation. This pragmatic, intellectually honest attitude is commendable.

What to Keep in Perspective

That said, readers should maintain measured judgment:

• All current tests are based on synthetic tasks (e.g., one-bit retention tasks). While these cleanly validate gradient propagation properties, they remain far removed from real-world language modeling, time-series forecasting, and other practical scenarios.
• Experiments were only run at small scale on CPU, lacking systematic validation on large datasets and real-world tasks.
• The "100% accuracy" extrapolation result is impressive, but on a well-designed synthetic task, a theoretically sound architecture should perform perfectly — this doesn't directly prove superiority in complex real-world settings.

A Unique Exploration in the RNN Revival Wave

This project is not an isolated effort. In recent years, with the rise of state space models and linear RNNs like Mamba and RWKV, both academia and industry have begun reexamining "attention-free" approaches to sequence modeling. These methods all seek to maintain long-sequence modeling capability while avoiding the quadratic complexity bottleneck of Transformers in long-context settings.

The novel-rnn-architectures project approaches this from the distinctive angle of harmonic-series weighting, contributing a new perspective to this line of exploration. While it remains at an early stage with limited experimental scale, its clear mathematical motivation, transparent open-source implementation, and candid approach to validation make it worth following.

For deep learning researchers and engineers, projects like this serve as a reminder: in an era of chasing large models and massive compute, returning to first principles and seeking breakthroughs through fundamental mathematical mechanisms still holds undeniable value. Whether this project can hold its own at large scale in real-world settings remains to be verified and extended by the community.