A New Take on AI Long-Term Memory: 50M Token Window, Faster and Cheaper Than Recomputation

The Memory Bottleneck in Large Language Models

Large language models can only process text that fits within their context window, and every time they receive a prompt, they recompute internal key-value states (KV state) from scratch. This repeated computation has two direct consequences: slower responses and skyrocketing GPU energy consumption. When a conversation or task involves millions of tokens of historical information, this cost becomes nearly unbearable.

A paper published on arXiv (arXiv:2610.10845v1) proposes a compelling idea — instead of recomputing every time, save the computed KV states and reload them precisely when needed. The open-source tool tested by the research team is called galahad-kv. Its core approach is to save the KV state of each ~16K-token chunk to an encrypted local NVMe disk, then reload it with byte-exact precision later, completely bypassing recomputation.

KV State (Key-Value State) is the core intermediate product of the attention mechanism in Transformer architectures. The self-attention computation in each layer generates a set of Key and Value vectors for every token in the input sequence — collectively known as the KV Cache. During inference, models typically retain these KV vectors in memory to avoid recomputing them for already-processed tokens. The problem is that KV cache size grows linearly with sequence length — for models with tens of billions of parameters, the KV cache for tens of thousands of tokens can already occupy dozens of gigabytes of VRAM. Scaling to millions or even tens of millions of tokens makes standard GPU memory completely inadequate. Traditional approaches either truncate history or recompute on every inference pass — both pay a price in either information loss or wasted compute. The core insight behind galahad-kv is to persist these intermediate products to disk, transforming "recompute every time" into "compute once, reuse persistently."

A Real-World Test at 50 Million Tokens

This research didn't stop at theoretical reasoning — it ran a large-scale test in a real-world scenario. The experimental setup used a single NVIDIA H100 with vLLM serving, testing Gemma 4 12B and Gemma 4 31B on 50 million tokens of real public text.

The results are impressive. Across every depth from 0 to 50 million tokens, every chunk probed by the researchers was successfully loaded back from encrypted storage with zero recomputation — 100 probes, 100 hits, for both models. This means the store-and-load mechanism remained stable across ultra-long sequences without degrading as depth increased.

Quantified Speed and Energy Advantages

Loading a chunk is 2.8 to 4.3× faster than recomputing it, while GPU energy consumption drops by 8.8 to 12.3×. More critically, GPU memory usage remained flat throughout the entire 50M-token streaming process. This addresses the most problematic aspect of long-context scenarios — VRAM bloat — because only one chunk needs to be loaded into VRAM at any given moment, rather than stuffing the entire history into the attention window.

Memory Recall Accuracy

The real test of whether long-term memory actually works is whether the model can answer factual questions planted millions of tokens earlier. In this test, the 12B model answered 82 out of 100 questions correctly, while the 31B model answered 98 out of 100. Notably, neither model fabricated answers — there were no hallucinated responses. This indicates that the mechanism preserves information fidelity, with accuracy differences primarily reflecting the models' own capabilities.

This Is Not a Wider Attention Window

The paper is quite candid about its own limitations, and understanding this is key to appreciating the work correctly.

First and foremost: this is state reuse, not a wider attention window. The system loads only one chunk at a time, and how well a question gets answered depends on the model itself. In other words, galahad-kv does not let the model "see" 50 million tokens simultaneously — it's more accurately described as an efficient external memory retrieval and state reuse mechanism.

Two real-world costs also deserve attention. First, writing to memory is a one-time cost — the initial processing of these texts still requires full KV state computation. Second, storage overhead is substantial: the KV state for 50 million tokens occupies multiple terabytes of local NVMe disk space. For scenarios requiring repeated access to the same large body of text, this upfront investment can be amortized through fast subsequent loads; for one-off tasks, the benefits are more limited.

Understanding the difference between an "attention window" and "external memory retrieval" helps clarify the value boundary of this work. Standard Transformer attention requires the model to "see" all relevant tokens in a single forward pass, with computational complexity scaling quadratically with sequence length — the fundamental bottleneck to extending context length. Recent industry efforts to expand attention windows (such as FlashAttention, sparse attention, and ring attention) essentially still aim to process longer sequences in a single inference pass. galahad-kv takes a fundamentally different path: it stores history in chunks and loads only the currently relevant chunk's KV state back into VRAM for computation. The model still effectively "sees" a limited window of content. This is closer in spirit to RAG (Retrieval-Augmented Generation) — the difference being that RAG retrieves raw text fragments and re-encodes them, whereas this method directly reuses already-encoded KV states, eliminating recomputation overhead, but also meaning that direct cross-chunk attention interactions don't exist.

Reproducibility and Benchmark Design

This research deserves credit on two methodological fronts. First, the test protocol is designed to resist common "benchmark gaming" tactics in long-context evaluations, aiming for a more genuine capability assessment. Long-context benchmarks have historically been easy to manipulate with various tricks, so a robust test protocol is valuable in its own right.

Second, the research provides a single-GPU reproducible setup using public software, with the galahad-kv package under a free license. This lowers the barrier for community verification and replication, and gives the approach a chance to be more broadly examined and adopted.

The problem of gaming long-context benchmarks has a long history in the research community. Common tactics include: placing key retrievable information at the beginning or end of a sequence (where models naturally pay more attention than to the middle — the "lost in the middle" phenomenon); using documents that appeared in the training set so the model can rely on memorization rather than true retrieval; and designing overly simple factual questions where semantic similarity search suffices. A genuinely rigorous long-context evaluation requires burying target information deep in the sequence, using text the model hasn't seen before, and demanding cross-passage reasoning rather than simple matching. The paper claims its test protocol is resistant to these techniques — a design choice that provides a more credible baseline for subsequent community validation.

Implications for AI Engineering

From an engineering perspective, this work offers a practical path for handling extremely long histories. Rather than attempting to overturn Transformer attention mechanics, it layers a persistent memory system on top of existing architecture — treating expensive KV computation results as reusable cached assets.

For applications that need to retain large amounts of context across sessions — customer service conversations, long document analysis, continuously running agents — this "compute once, use many times" pattern has clear potential. Of course, the terabyte-scale storage requirements and the "one chunk at a time" limitation mean this isn't a universal solution, but rather a cost-effective choice for specific scenarios. As NVMe storage costs continue to fall, this kind of storage-for-compute trade-off may play an increasingly important role in long-term memory AI systems.