ILRN
A recurrent neural network architecture that replaces the self-attention mechanism with an inverse-latency memory mechanism, assigning a 1/k decay weight to historical inputs to achieve direct access to history.
Timeline (last 90 days)
ILRN (Inverse Lag Recurrent Network) is a project proposed by developer rickey1990 that attempts to replace the Transformer's self-attention mechanism with an Inverse Lag Recurrent Memory (ILRM) mechanism.
The recurrent part of ILRN adopts a dual-gating design: the access gate controls the contribution of the previous recurrent memory to the candidate state, and the write gate determines how much of the candidate state is updated into the stored memory.
In next-word prediction comparison experiments on the CNN/DailyMail dataset, ILRN and Transformer each had about 532K parameters, were each trained on 20 million next-word targets, and were trained independently under two context windows of 96 and 256.
Under a 96-token context, ILRN's held-out test set top-1 accuracy was 27.17%, and Transformer's was 25.83%.
The author himself states that these are preliminary, small-scale experiments with limited training rounds and cannot serve as evidence that ILRN is generally superior to Transformer.
ILRN's initial CPU implementation was clearly slower than Transformer in training efficiency, and the author later tried a fused CPU kernel to improve execution speed while keeping ILRM's mathematical logic unchanged.
The author has uploaded the research PDF, architecture formulas, Python implementation, fused kernel, and validation tests to GitHub, with the repository address being github.com/rickey1990/ILRN.
All Facts (7)
ILRN (Inverse Lag Recurrent Network) is a project proposed by developer rickey1990 that attempts to replace the Transformer's self-attention mechanism with an Inverse Lag Recurrent Memory (ILRM) mechanism.
50%UnverifiedThe recurrent part of ILRN adopts a dual-gating design: the access gate controls the contribution of the previous recurrent memory to the candidate state, and the write gate determines how much of the candidate state is updated into the stored memory.
50%UnverifiedIn next-word prediction comparison experiments on the CNN/DailyMail dataset, ILRN and Transformer each had about 532K parameters, were each trained on 20 million next-word targets, and were trained independently under two context windows of 96 and 256.
50%UnverifiedUnder a 96-token context, ILRN's held-out test set top-1 accuracy was 27.17%, and Transformer's was 25.83%.
50%UnverifiedThe author himself states that these are preliminary, small-scale experiments with limited training rounds and cannot serve as evidence that ILRN is generally superior to Transformer.
50%UnverifiedILRN's initial CPU implementation was clearly slower than Transformer in training efficiency, and the author later tried a fused CPU kernel to improve execution speed while keeping ILRM's mathematical logic unchanged.
50%UnverifiedThe author has uploaded the research PDF, architecture formulas, Python implementation, fused kernel, and validation tests to GitHub, with the repository address being github.com/rickey1990/ILRN.
50%