[KongchangAI]
· 2 min read· 1,227 words

LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

LoopFill fuses suffix drafts with shallow loop predictions into a single draft tree, accelerating looped Transformer decoding by up to 5.85x.

LoopFill is a training-free speculative decoding algorithm for looped Transformers that addresses high latency from sequential loops during autoregressive decoding. It combines multi-token suffix drafts with shallow loop predictions into a single draft tree, letting suffix lookup immediately extend shallow predictions so multiple future tokens enter the pipeline at once; a shared block computes top candidates across loop depths in one batch, and full-depth verification preserves the original output distribution. It achieves 5.85x and 5.13x average speedups on Ouro-2.6B and Huginn-3.5B, outperforming LoopAhead by 40% and 25%.

Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers, but during autoregressive decoding, each token requires multiple sequential passes through the model, resulting in high decoding latency. Existing pipelined self-speculative methods leverage shallow loop predictions to process tokens at different loop depths simultaneously, but each token must reach the draft depth before the next token can enter the pipeline, limiting parallelism. LoopFill is a training-free speculative decoding algorithm that innovatively combines multi-token suffix drafting with shallow loop predictions into a single draft tree, allowing suffix lookup to immediately extend shallow predictions so that multiple future tokens enter the pipeline simultaneously. A shared block computes the highest-scoring draft candidates across different loop depths in a single batch, while full-depth verification preserves the model's autoregressive output distribution. On seven benchmarks, LoopFill achieves average speedups of 5.85x and 5.13x over greedy autoregressive decoding on Ouro-2.6B and Huginn-3.5B, respectively, outperforming the strongest baseline LoopAhead by 40% and 25%.

LoopFill diagram

The Decoding Bottleneck of Looped Transformers

Looped Transformers (such as the Ouro and Huginn series) reduce parameter count by recycling the same set of layers, but at the cost of requiring multiple sequential executions during inference. Decoding each token requires multiple rounds of looping, significantly increasing latency. Speculative decoding accelerates generation through draft models, but traditional methods require independent draft models, whereas looped Transformers inherently possess intermediate predictions at different loop depths that can be used for self-speculation. However, in existing pipelined methods, tokens must enter the pipeline one by one, failing to fully exploit the parallel potential offered by the loop structure.

Loop depth refers to the number of times a token must pass through the same set of layers in a looped Transformer. For example, if a model is configured during training to perform 4 loops, the hidden state of each token passes through this set of layers 4 times before producing the final output. Greater loop depth means higher effective computation for the model, but the serial latency per token during decoding also increases linearly. Shallow loop prediction exploits intermediate layer outputs at smaller loop depths—such as hidden states that have only undergone 1 or 2 loops—as low-cost drafts, because these intermediate predictions already encode partial context information while requiring only a few layers of computation. Understanding the relationship between loop depth and shallow prediction is a prerequisite for understanding how LoopFill unifies candidates at different depths into a draft tree.

Core Mechanism of LoopFill

The key innovation of LoopFill lies in unifying suffix drafts and shallow loop predictions into a single draft tree. Suffix drafts use n-gram matching to rapidly generate multiple candidate tokens, while shallow predictions provide low-cost drafts based on the current loop depth. Combined, a suffix lookup can immediately extend shallow predictions, allowing multiple future tokens to enter the pipeline simultaneously. A shared block computes the highest-scoring draft candidates across different loop depths in a single batch, and full-depth verification ensures that the output distribution matches the original autoregressive model. The entire process requires no additional training and can be directly applied to existing looped LLMs.

Suffix draft is a speculative decoding technique based on n-gram matching: it extracts a short suffix from the end of the currently generated token sequence and looks up subsequent token sequences beginning with that suffix in a pre-built n-gram index or training corpus, thereby proposing multiple candidate tokens at once. This approach does not rely on additional neural networks and has extremely low cost, but its hit rate depends on text repetitiveness. LoopFill's innovation is that suffix drafts no longer work independently but instead connect with shallow loop predictions: after a shallow prediction yields an initial candidate token, suffix lookup immediately uses that candidate as an anchor to extend a longer candidate sequence, allowing the draft tree to grow in both depth and width simultaneously. In this way, multiple future tokens can enter the pipeline almost simultaneously, breaking through the limitation of prior pipelined self-speculative methods where tokens must wait individually for draft depth.

Draft tree is a structure for organizing multiple candidate tokens in speculative decoding. Unlike single-chain drafts, tree-structured drafts allow multiple branch candidates to be retained at each position, covering more possible continuation paths in a single verification. During verification, the model computes probabilities for all nodes in the tree in parallel and accepts the longest possible prefix along the highest-probability path. LoopFill merges multiple candidate chains produced by suffix drafts with candidates from shallow loop predictions into the same tree, and the shared block then ranks and selects the highest-scoring nodes at different loop depths, so that the verification batch simultaneously contains candidates from different depths and different suffix matches. The quality of the draft tree directly determines acceptance length, while full-depth verification ensures that accepted tokens still follow the original autoregressive distribution.

Performance

The research team evaluated LoopFill on seven benchmarks. On Ouro-2.6B, the average speedup reaches 5.85x, and on Huginn-3.5B it reaches 5.13x. Compared with the strongest baseline LoopAhead, LoopFill improves by 40% and 25%, respectively. These results demonstrate that decoding latency can be substantially reduced by more fully exploiting the intermediate predictions and pipeline parallelism of looped Transformers.

Average speedup ratio is typically defined as the ratio of the average per-token latency of greedy autoregressive decoding to that of the speculative decoding method, or the ratio of total decoding time at a fixed output length. This metric is jointly affected by acceptance length, verification overhead, and draft cost. The 5.85x and 5.13x speedups reported by LoopFill are averaged over seven benchmarks, meaning that speedups fluctuate across tasks: tasks with high suffix-match hit rates (such as code or formatted text) typically show more pronounced speedups, while highly free-form generation may show lower ones. The comparison with LoopAhead uses the same models and hardware conditions, and the relative improvements of 40% and 25% mainly come from increased pipeline parallelism rather than changes in verification accuracy.

Multi-Source Cross-Validation

The analysis in this article is based on the paper abstract and publicly available diagrams. According to interpretations by the paper authors and Bilibili creators, LoopFill's training-free nature makes it easy to deploy, while full-depth verification ensures generation quality. At the same time, a YouTube channel noted that the method is primarily applicable to looped architectures, and its generality to standard Transformers still requires further validation. Overall, LoopFill offers a new approach for efficient inference of looped LLMs, but compatibility and scalability in practical applications deserve attention.

One-line Summary

LoopFill fuses suffix drafts with shallow loop predictions into a single draft tree, accelerating looped Transformer decoding by up to 5.85x.

Paragraph Summary

LoopFill is a training-free speculative decoding algorithm for looped Transformers, designed to address the high latency caused by multiple sequential loops during autoregressive decoding. Existing pipelined self-speculative methods require each token to reach draft depth before the next token can enter the pipeline, limiting parallelism. LoopFill combines multi-token suffix drafting with shallow loop predictions into a single draft tree, enabling suffix lookup to immediately extend shallow predictions so that multiple future tokens enter the pipeline simultaneously; a shared block computes the highest-scoring candidates across different loop depths in a single batch, while full-depth verification preserves the original autoregressive output distribution. On seven benchmarks, LoopFill achieves average speedups of 5.85x and 5.13x on Ouro-2.6B and Huginn-3.5B, respectively, outperforming the strongest baseline LoopAhead by 40% and 25%.

#LoopFill#speculative decoding#looped Transformer#pipeline parallelism#suffix draft#draft tree#training-free#LLM inference acceleration
Share:

Related articles