NoteBank
A training-free sparse episodic memory add-on module that improves long-context retrieval recall for linear attention and state space models by storing and replaying raw text segments, without modifying the backbone network weights.
Timeline (last 90 days)
KAUST's paper "History Replays Itself" (arXiv:2610.15715) proposes NoteBank, a training-free add-on module that improves long-context recall by storing raw text fragments and re-feeding them to the model when a question arrives.
NoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
On six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
NoteBank's persistent storage is approximately 1/2600 of a full multi-head attention KV cache of equivalent width, depth, and precision (74.5 MiB versus 192 GiB in the paper's Figure 1).
NoteBank's additional compute is at most 0.025% of the analysis FLOPs.
NoteBank's writing stage takes place during context encoding, before any question appears, selecting notes based on each token's key vector uniqueness (1 minus the cosine similarity between the token's key and the block's average key, then averaged over layers and heads), and cannot foresee what future questions will need.
In NoteBank's addressing stage, each note stores a compact address for each layer-head channel (the native query vector at the note's most unique token); when a question arrives, each channel independently computes cosine similarity and votes, and the top R notes with the most votes are retrieved, with earlier notes preferred in case of ties.
In NoteBank's replay stage, the retrieved notes are concatenated in original document order and reprocessed starting from the unmodified backbone network (starting from a saved copy of the state), after which the question is processed and the answer generated.
NoteBank's core retrieval results require no training, but the paper optionally trains a lightweight LoRA adapter to help the model exploit replayed evidence in downstream tasks, with the adapter enabled only during the replay and answering stages.
On LongBench v1/v2 and MemoryAgentBench, NoteBank also reported improvements, covering everything from three 1.3B research models up to Mamba-2-8B and a trained Qwen3.5-9B.
3 more timeline events
All Facts (13)
When the information a question needs is sparsely scattered throughout the document, or requires aggregating multiple paragraphs, or is not salient during the write stage, NoteBank's sparse-archive recall drops noticeably, which is one of the limitations the paper acknowledges.
50%UnverifiedNoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
50%UnverifiedOn six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
50%UnverifiedNoteBank's persistent storage is approximately 1/2600 of a full multi-head attention KV cache of equivalent width, depth, and precision (74.5 MiB versus 192 GiB in the paper's Figure 1).
50%UnverifiedNoteBank's additional compute is at most 0.025% of the analysis FLOPs.
50%UnverifiedNoteBank's writing stage takes place during context encoding, before any question appears, selecting notes based on each token's key vector uniqueness (1 minus the cosine similarity between the token's key and the block's average key, then averaged over layers and heads), and cannot foresee what future questions will need.
50%UnverifiedIn NoteBank's addressing stage, each note stores a compact address for each layer-head channel (the native query vector at the note's most unique token); when a question arrives, each channel independently computes cosine similarity and votes, and the top R notes with the most votes are retrieved, with earlier notes preferred in case of ties.
50%UnverifiedIn NoteBank's replay stage, the retrieved notes are concatenated in original document order and reprocessed starting from the unmodified backbone network (starting from a saved copy of the state), after which the question is processed and the answer generated.
50%UnverifiedNoteBank's core retrieval results require no training, but the paper optionally trains a lightweight LoRA adapter to help the model exploit replayed evidence in downstream tasks, with the adapter enabled only during the replay and answering stages.
50%UnverifiedOn LongBench v1/v2 and MemoryAgentBench, NoteBank also reported improvements, covering everything from three 1.3B research models up to Mamba-2-8B and a trained Qwen3.5-9B.
50%UnverifiedNoteBank's storage and compute claims are analytical: the paper's FLOP numbers are layer-level analytical estimates, and the authors state that they do not measure runtime, and that exhaustive matching and replay add actual latency.
50%UnverifiedNoteBank is a preprint, and its results have not yet been peer-reviewed or independently reproduced.
50%UnverifiedKAUST's paper "History Replays Itself" (arXiv:2610.15715) proposes NoteBank, a training-free add-on module that improves long-context recall by storing raw text fragments and re-feeding them to the model when a question arrives.
50%