LoopSpec
A self-speculative decoding method for recurrent Transformers that uses predictions from intermediate loop steps as the draft source, parallelizing drafting and verification across loops within a single step.
Timeline (last 90 days)
WaveFront Decoding (WFD) and LoopSpec are self-speculative decoding methods that use a wavefront-style pipeline to draft and target tokens across loops within a single step.
Two key flaws of WFD and LoopSpec are: drafts are generated only at fixed loop points while ignoring intermediate prediction changes; and when batching tokens across loops, each step must load up to R times the KV cache.
Experimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
All Facts (3)
WaveFront Decoding (WFD) and LoopSpec are self-speculative decoding methods that use a wavefront-style pipeline to draft and target tokens across loops within a single step.
50%UnverifiedTwo key flaws of WFD and LoopSpec are: drafts are generated only at fixed loop points while ignoring intermediate prediction changes; and when batching tokens across loops, each step must load up to R times the KV cache.
50%UnverifiedExperimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
50%