[KongchangAI]
ConceptLoopCache框架

LoopCache

A self-speculative decoding acceleration framework for long-context recurrent Transformers. Through confidence-guided dynamic draft updates and a cyclic rotating KV cache strategy, it reduces the KV cache memory traffic caused by cross-loop batching.

Timeline (last 90 days)

Nov 19

LoopCache introduces two core techniques: confidence-guided dynamic draft updating and a loop KV cache strategy.

Unverified50%
Nov 19

LoopCache uses confidence signals to dynamically refine drafts across all intermediate loops, rather than generating drafts only at fixed loop points.

Unverified50%
Nov 19

LoopCache accesses the loop-specific KV caches in a cyclic round-robin manner, loading only a single loop's KV cache at each step and limiting per-step KV traffic to the single-loop level.

Unverified50%
Nov 19

For tokens requiring batching, LoopCache batches on the most recently matched loop cycle to guarantee lossless verification.

Unverified50%
Nov 19

Experimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.

Unverified50%
Nov 19

By limiting per-step KV traffic to the single-loop level while using confidence guidance to maintain draft quality, LoopCache strikes a balance between parallelization gains and memory traffic.

Unverified50%

All Facts (6)

Source Articles