CopyLoop
A training-free decoding acceleration method for recurrent Transformers. It copies candidate suffixes from the context, screens them at shallow recurrent depths, and uses continuous recurrence scheduling to reduce the number of full-depth computations, thereby improving decoding speed while keeping the output distribution consistent.
Timeline (last 90 days)
CopyLoop is a training-free decoding acceleration algorithm that accelerates decoding of recurrent Transformers by combining low-cost context-copy proposals, shallow-layer screening, and continuous recurrent scheduling.
CopyLoop's workflow is divided into two stages, proposal and screening: it first copies multiple candidate suffixes from the context, then rapidly evaluates the candidates at shallow recurrent depths.
CopyLoop's shallow-layer screening exploits the anytime property of recurrent Transformers, scoring copy candidates at low depth, eliminating low-probability candidates, and retaining a few to enter full-depth verification.
CopyLoop's continuous recurrent scheduling treats recurrent depth as overlapping pipeline stages, allowing later blocks to begin processing based on shallow-layer results while earlier blocks are still advancing.
The paper's ablation experiments confirm a strong synergy between shallow-layer screening and continuous recurrent scheduling, with either technique used alone being less effective than the combination of both.
The research team evaluated CopyLoop on six checkpoints from three model families, Ouro, Raven, and Huginn, with tasks covering code generation and multi-step mathematical reasoning.
On the Base models, CopyLoop achieved decoding speedups of 2.98x to 10.70x compared with full-depth autoregressive decoding.
On the Thinking models, CopyLoop achieved decoding speedups of 2.74x to 3.52x compared with full-depth autoregressive decoding.
On all evaluated tasks, CopyLoop either outperformed all competing baselines or performed very close to them.
CopyLoop requires only lightweight model calibration and no retraining.
3 more timeline events
All Facts (13)
CopyLoop is a training-free decoding acceleration algorithm that accelerates decoding of recurrent Transformers by combining low-cost context-copy proposals, shallow-layer screening, and continuous recurrent scheduling.
50%UnverifiedCopyLoop's workflow is divided into two stages, proposal and screening: it first copies multiple candidate suffixes from the context, then rapidly evaluates the candidates at shallow recurrent depths.
50%UnverifiedCopyLoop's shallow-layer screening exploits the anytime property of recurrent Transformers, scoring copy candidates at low depth, eliminating low-probability candidates, and retaining a few to enter full-depth verification.
50%UnverifiedCopyLoop's continuous recurrent scheduling treats recurrent depth as overlapping pipeline stages, allowing later blocks to begin processing based on shallow-layer results while earlier blocks are still advancing.
50%UnverifiedThe paper's ablation experiments confirm a strong synergy between shallow-layer screening and continuous recurrent scheduling, with either technique used alone being less effective than the combination of both.
50%UnverifiedThe research team evaluated CopyLoop on six checkpoints from three model families, Ouro, Raven, and Huginn, with tasks covering code generation and multi-step mathematical reasoning.
50%UnverifiedOn the Base models, CopyLoop achieved decoding speedups of 2.98x to 10.70x compared with full-depth autoregressive decoding.
50%UnverifiedOn the Thinking models, CopyLoop achieved decoding speedups of 2.74x to 3.52x compared with full-depth autoregressive decoding.
50%UnverifiedOn all evaluated tasks, CopyLoop either outperformed all competing baselines or performed very close to them.
50%UnverifiedCopyLoop requires only lightweight model calibration and no retraining.
50%UnverifiedIn scenarios with long generation lengths and reusable context, CopyLoop's speedup is more pronounced.
50%UnverifiedCopyLoop's speedup depends on the presence of reusable segments in the context; for fully open-domain generation tasks, the proposal hit rate may decline.
50%UnverifiedThe threshold setting for shallow-layer screening may need to be tuned for different models, which is exactly the problem the lightweight calibration mentioned in the paper is meant to solve.
50%