KVMem
A KV cache offloading tool that transfers the key-value cache generated during Transformer inference from GPU memory to system RAM, thereby freeing VRAM to support the execution of larger models.
Timeline (last 90 days)
Combining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.
Looking at each component individually, the performance gain is only about 10%–15%, but combining all three produces a notable synergistic effect (nonlinear acceleration).
KVMem's role is to offload the KV cache pool from VRAM to system memory, thereby freeing up VRAM.
KVMem and Ninfer originally had conflicts in encoding and cache handling and were mutually incompatible; the new version resolves this compatibility issue.
Previously, running a regular model with llama.cpp combined with KVMem could achieve roughly 100-plus tokens/s.
All Facts (5)
Combining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.
50%UnverifiedLooking at each component individually, the performance gain is only about 10%–15%, but combining all three produces a notable synergistic effect (nonlinear acceleration).
50%UnverifiedKVMem's role is to offload the KV cache pool from VRAM to system memory, thereby freeing up VRAM.
50%UnverifiedKVMem and Ninfer originally had conflicts in encoding and cache handling and were mutually incompatible; the new version resolves this compatibility issue.
50%UnverifiedPreviously, running a regular model with llama.cpp combined with KVMem could achieve roughly 100-plus tokens/s.
50%