galahad-kv
An open-source KV state persistence tool that saves the KV cache generated during Transformer inference to encrypted local NVMe storage. It supports byte-precise reloading to avoid redundant computation, substantially accelerating long-context inference and reducing GPU energy consumption.
Timeline (last 90 days)
The open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
At various depths from 0 to 50 million tokens, every chunk probed by the researchers was successfully loaded back from encrypted storage, achieving 100 hits out of 100 probes for both models.
Loading a chunk is 2.8 to 4.3 times faster than recomputing it, and GPU energy consumption is reduced by 8.8 to 12.3 times.
Throughout the entire 50-million-token stream processing, GPU memory usage remained stable because only one chunk needs to be loaded into memory at any given time.
galahad-kv reuses stored states rather than widening the attention window; the system loads only one chunk at a time and does not allow the model to see 50 million tokens simultaneously.
Writing to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
The galahad-kv approach resembles the RAG paradigm, with the difference that RAG retrieves raw text segments and re-encodes them, whereas this method directly reuses already-encoded KV states, but direct attention interactions across chunks do not exist.
The research provides a single-GPU reproducible solution using publicly available software, and the galahad-kv package is released under a free license.
The study's testing protocol is designed to resist common long-context benchmark gaming tactics, such as placing key information at the beginning and end of a sequence, using training-set documents, or designing overly simple questions.
All Facts (9)
The open-source tool galahad-kv saves the KV states of each approximately 16,000-token chunk to an encrypted local NVMe disk, then reloads them byte-accurately, completely skipping recomputation.
50%UnverifiedAt various depths from 0 to 50 million tokens, every chunk probed by the researchers was successfully loaded back from encrypted storage, achieving 100 hits out of 100 probes for both models.
50%UnverifiedLoading a chunk is 2.8 to 4.3 times faster than recomputing it, and GPU energy consumption is reduced by 8.8 to 12.3 times.
50%UnverifiedThroughout the entire 50-million-token stream processing, GPU memory usage remained stable because only one chunk needs to be loaded into memory at any given time.
50%Unverifiedgalahad-kv reuses stored states rather than widening the attention window; the system loads only one chunk at a time and does not allow the model to see 50 million tokens simultaneously.
50%UnverifiedWriting to memory is a one-time cost, as the first processing of text still requires full KV state computation; the KV states for 50 million tokens occupy several terabytes of local NVMe disk space.
50%UnverifiedThe galahad-kv approach resembles the RAG paradigm, with the difference that RAG retrieves raw text segments and re-encodes them, whereas this method directly reuses already-encoded KV states, but direct attention interactions across chunks do not exist.
50%UnverifiedThe research provides a single-GPU reproducible solution using publicly available software, and the galahad-kv package is released under a free license.
50%UnverifiedThe study's testing protocol is designed to resist common long-context benchmark gaming tactics, such as placing key information at the beginning and end of a sequence, using training-set documents, or designing overly simple questions.
50%