CacheGraft
A retrieval-augmented generation (RAG) method built on the Cache-to-Cache (C2C) fusion framework. It encodes documents offline into fused KV caches between a large model and a small model, then injects them directly into the small model at inference time, achieving knowledge transfer without invoking the large model.
Timeline (last 90 days)
CacheGraft is a retrieval-augmented generation method based on the Cache-to-Cache (C2C) fusion framework, which in an offline stage encodes each passage in the corpus into a fused KV cache representation between a large model and a small model, and directly injects it into the small model at query time without calling the large model.
The receiver small model used in the CacheGraft experiments has 0.6B parameters, and the source large model has 4B parameters.
CacheGraft was evaluated on five question-answering benchmarks, using two independently running local judge models.
CacheGraft outperformed the plain-text receiver baseline in 29 of 30 comparisons, of which 17 were statistically significant.
CacheGraft's gains are zero-shot for the fuser, which was never trained on retrieval data.
The CacheGraft paper positions itself as 'preliminary results' and points out that the compatibility of cache injection, the transfer effectiveness between different model pairs, and the scalability of cache representations as the corpus size grows are all open problems.
All Facts (6)
CacheGraft is a retrieval-augmented generation method based on the Cache-to-Cache (C2C) fusion framework, which in an offline stage encodes each passage in the corpus into a fused KV cache representation between a large model and a small model, and directly injects it into the small model at query time without calling the large model.
50%UnverifiedThe receiver small model used in the CacheGraft experiments has 0.6B parameters, and the source large model has 4B parameters.
50%UnverifiedCacheGraft was evaluated on five question-answering benchmarks, using two independently running local judge models.
50%UnverifiedCacheGraft outperformed the plain-text receiver baseline in 29 of 30 comparisons, of which 17 were statistically significant.
50%UnverifiedCacheGraft's gains are zero-shot for the fuser, which was never trained on retrieval data.
50%UnverifiedThe CacheGraft paper positions itself as 'preliminary results' and points out that the compatibility of cache injection, the transfer effectiveness between different model pairs, and the scalability of cache representations as the corpus size grows are all open problems.
50%