PLUG
A novel RNN architecture that replaces traditional exponential decay with polynomial decay driven by harmonic numbers, addressing the vanishing gradient problem in long-sequence modeling. It operates without an attention mechanism and supports FFT-accelerated computation.
Timeline (last 90 days)
A developer published an open-source project named novel-rnn-architectures on Reddit and GitHub, proposing two novel RNN architectures called PLUG and ILRM.
The core objective of PLUG and ILRM is to solve the vanishing gradient problem in RNN long-sequence modeling without relying on attention modules.
The project feeds a deterministic, normalized inverse-lag history vector P_t into the recurrent unit, weighted by harmonic numbers, causing historical information to decay at a rate of 1/k.
PLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
On 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
All Facts (5)
The core objective of PLUG and ILRM is to solve the vanishing gradient problem in RNN long-sequence modeling without relying on attention modules.
50%UnverifiedThe project feeds a deterministic, normalized inverse-lag history vector P_t into the recurrent unit, weighted by harmonic numbers, causing historical information to decay at a rate of 1/k.
50%UnverifiedPLUG and ILRM were trained only on 16–64 step delays, successfully extrapolating to 4096 steps, achieving 100% accuracy on a one-bit information retention task, while the baseline standard GRU degraded to random-guess level.
50%UnverifiedOn 1024-step sequences, the average gradient of the initial token for PLUG and ILRM remains stably at around the 10⁻⁴ order of magnitude, while the standard GRU gradient drops as low as approximately 10⁻¹⁴⁵, resulting in numerical underflow.
50%UnverifiedA developer published an open-source project named novel-rnn-architectures on Reddit and GitHub, proposing two novel RNN architectures called PLUG and ILRM.
50%