[KongchangAI]
Concept

LoopSpec

A self-speculative decoding method for recurrent Transformers that uses predictions from intermediate loop steps as the draft source, parallelizing drafting and verification across loops within a single step.

Timeline (last 90 days)

Nov 19

WaveFront Decoding (WFD) and LoopSpec are self-speculative decoding methods that use a wavefront-style pipeline to draft and target tokens across loops within a single step.

Unverified50%
Nov 19

Two key flaws of WFD and LoopSpec are: drafts are generated only at fixed loop points while ignoring intermediate prediction changes; and when batching tokens across loops, each step must load up to R times the KV cache.

Unverified50%
Nov 19

Experimental results show that LoopCache achieves average speedups of 10.25x and 14.56x over WFD and LoopSpec, respectively.

Unverified50%

All Facts (3)

Source Articles