BeNice!
A post-training method that broadcasts the KV cache of the last loop to the first loop and trains compatible residuals, enabling KV cache sharing across loops in recurrent architectures and reducing inference memory usage without loss of quality.
Timeline (last 90 days)
The new paper proposes a two-stage scheme: pretraining uses random depth to induce contractive residuals, and post-training uses BeNice! to broadcast the KV cache and achieve KV sharing.
The BeNice! method broadcasts the KV cache from the last loop to the first loop, and trains the first loop to produce residuals compatible with it.
While enabling KV sharing, BeNice! matches standard fine-tuning (without sharing) in validation perplexity, without sacrificing the original model's quality.
All Facts (3)
The new paper proposes a two-stage scheme: pretraining uses random depth to induce contractive residuals, and post-training uses BeNice! to broadcast the KV cache and achieve KV sharing.
50%UnverifiedThe BeNice! method broadcasts the KV cache from the last loop to the first loop, and trains the first loop to produce residuals compatible with it.
50%UnverifiedWhile enabling KV sharing, BeNice! matches standard fine-tuning (without sharing) in validation perplexity, without sacrificing the original model's quality.
50%