WaveFront Decoding
A self-speculative decoding method for recurrent Transformers that uses a wavefront-style pipeline to draft and target tokens across loops within a single step, requiring no additional draft model.
Timeline (last 90 days)
WaveFront Decoding (WFD) and LoopSpec are self-speculative decoding methods that use a wavefront-style pipeline to draft and target tokens across loops within a single step.
Two key flaws of WFD and LoopSpec are: drafts are generated only at fixed loop points while ignoring intermediate prediction changes; and when batching tokens across loops, each step must load up to R times the KV cache.
Experimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
All Facts (3)
WaveFront Decoding (WFD) and LoopSpec are self-speculative decoding methods that use a wavefront-style pipeline to draft and target tokens across loops within a single step.
50%UnverifiedTwo key flaws of WFD and LoopSpec are: drafts are generated only at fixed loop points while ignoring intermediate prediction changes; and when batching tokens across loops, each step must load up to R times the KV cache.
50%UnverifiedExperimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.
50%