ILRM
A novel RNN architecture belonging to the same novel-rnn-architectures project as PLUG. It employs a harmonic-number weighting mechanism to address the vanishing gradient problem, achieving long-sequence modeling capability without relying on attention modules.
Timeline (last 90 days)
ILRN's initial CPU implementation was clearly slower than Transformer in training efficiency, and the author later tried a fused CPU kernel to improve execution speed while keeping ILRM's mathematical logic unchanged.
ILRN (Inverse Lag Recurrent Network) is a project proposed by developer rickey1990 that attempts to replace the Transformer's self-attention mechanism with an Inverse Lag Recurrent Memory (ILRM) mechanism.
ILRM assigns a weight of 1/k to each input before the current position according to its distance k, so that closer inputs contribute more, earlier inputs continue to participate in the computation with decreasing weights, and a normalized historical representation is finally obtained.
ILRM's inverse lag history computation is accelerated with causal FFT convolution, but the recurrent state update is still serial and cannot be parallelized.
ILRN-4-96 is formed by stacking four independent ILRM blocks in sequence, each block having its own inverse lag history, two gates, recurrent state, and learnable parameters, with pre/post LayerNorm and residual connections between blocks.
A developer published an open-source project named novel-rnn-architectures on Reddit and GitHub, proposing two novel RNN architectures called PLUG and ILRM.
The core objective of PLUG and ILRM is to solve the vanishing gradient problem in RNN long-sequence modeling without relying on attention modules.
The project feeds a deterministic, normalized inverse-lag history vector P_t into the recurrent unit, weighted by harmonic numbers, causing historical information to decay at a rate of 1/k.
PLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
On 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
All Facts (10)
ILRN's initial CPU implementation was clearly slower than Transformer in training efficiency, and the author later tried a fused CPU kernel to improve execution speed while keeping ILRM's mathematical logic unchanged.
50%UnverifiedILRN (Inverse Lag Recurrent Network) is a project proposed by developer rickey1990 that attempts to replace the Transformer's self-attention mechanism with an Inverse Lag Recurrent Memory (ILRM) mechanism.
50%UnverifiedILRM assigns a weight of 1/k to each input before the current position according to its distance k, so that closer inputs contribute more, earlier inputs continue to participate in the computation with decreasing weights, and a normalized historical representation is finally obtained.
50%UnverifiedILRM's inverse lag history computation is accelerated with causal FFT convolution, but the recurrent state update is still serial and cannot be parallelized.
50%UnverifiedILRN-4-96 is formed by stacking four independent ILRM blocks in sequence, each block having its own inverse lag history, two gates, recurrent state, and learnable parameters, with pre/post LayerNorm and residual connections between blocks.
50%UnverifiedThe core objective of PLUG and ILRM is to solve the vanishing gradient problem in RNN long-sequence modeling without relying on attention modules.
50%UnverifiedThe project feeds a deterministic, normalized inverse-lag history vector P_t into the recurrent unit, weighted by harmonic numbers, causing historical information to decay at a rate of 1/k.
50%UnverifiedPLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
50%UnverifiedOn 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
50%UnverifiedA developer published an open-source project named novel-rnn-architectures on Reddit and GitHub, proposing two novel RNN architectures called PLUG and ILRM.
50%