LoopCache
A self-speculative decoding acceleration framework for long-context recurrent Transformers. Through confidence-guided dynamic draft updates and a cyclic rotating KV cache strategy, it reduces the KV cache memory traffic caused by cross-loop batching.
Timeline (last 90 days)
LoopCache introduces two core techniques: confidence-guided dynamic draft updating and a loop KV cache strategy.
LoopCache uses confidence signals to dynamically refine drafts across all intermediate loops, rather than generating drafts only at fixed loop points.
LoopCache accesses the loop-specific KV caches in a cyclic round-robin manner, loading only a single loop's KV cache at each step and limiting per-step KV traffic to the single-loop level.
For tokens requiring batching, LoopCache batches on the most recently matched loop cycle to guarantee lossless verification.
Experimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
By limiting per-step KV traffic to the single-loop level while using confidence guidance to maintain draft quality, LoopCache strikes a balance between parallelization gains and memory traffic.
All Facts (6)
LoopCache introduces two core techniques: confidence-guided dynamic draft updating and a loop KV cache strategy.
50%UnverifiedLoopCache uses confidence signals to dynamically refine drafts across all intermediate loops, rather than generating drafts only at fixed loop points.
50%UnverifiedLoopCache accesses the loop-specific KV caches in a cyclic round-robin manner, loading only a single loop's KV cache at each step and limiting per-step KV traffic to the single-loop level.
50%UnverifiedFor tokens requiring batching, LoopCache batches on the most recently matched loop cycle to guarantee lossless verification.
50%UnverifiedExperimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
50%UnverifiedBy limiting per-step KV traffic to the single-loop level while using confidence guidance to maintain draft quality, LoopCache strikes a balance between parallelization gains and memory traffic.
50%