fast weight matrix
A weight matrix updated online according to the input stream during inference, without backpropagation or gradient descent. It stores key-value associations across windows, enabling associative recall under constant memory.
Timeline (last 90 days)
The fast weight matrix stores key-value associations that have appeared across windows (episodic key-value memory), not a compressed distributed context.
On cross-window, cross-session associative recall tasks, attention mechanisms and LoRA achieve a recall rate of 0.000 by construction, while the fast weight matrix scores 0.62/0.43/0.62 across three seed runs, averaging 0.56; removing the matrix collapses recall to 0.008.
The awareliquid experiment demonstrates that the fast weight matrix and its accompanying state can be serialized to disk across processes and restored bit-exact losslessly, with recall results after restoration identical bit-for-bit to the source process.
The fast weight matrix does not bring understanding, reasoning or emotion, nor does it improve long-context language modeling performance, and the memory has no redundant backup.
All Facts (4)
The fast weight matrix stores key-value associations that have appeared across windows (episodic key-value memory), not a compressed distributed context.
50%UnverifiedOn cross-window, cross-session associative recall tasks, attention mechanisms and LoRA achieve a recall rate of 0.000 by construction, while the fast weight matrix scores 0.62/0.43/0.62 across three seed runs, averaging 0.56; removing the matrix collapses recall to 0.008.
50%UnverifiedThe awareliquid experiment demonstrates that the fast weight matrix and its accompanying state can be serialized to disk across processes and restored bit-exact losslessly, with recall results after restoration identical bit-for-bit to the source process.
50%UnverifiedThe fast weight matrix does not bring understanding, reasoning or emotion, nor does it improve long-context language modeling performance, and the memory has no redundant backup.
50%