Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping — One Model, Five Budgets

Large language models need to serve applications with different inference budgets, but training and maintaining a separate model for each budget is prohibitively expensive. A recent paper titled Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping proposes Accordion, a Mixture-of-Experts (MoE)-based model that supports five inference budgets through fixed-prefix layer looping. It achieves an average improvement of 3.01 percentage points across 37 tasks, and the middle budget captures 89% of the gain using only half the additional layer executions.

Background: The Inference Budget Dilemma and the Limits of Looped Transformers

Different applications have different requirements for inference latency, compute resources, and cost, so large models need to flexibly adjust the amount of computation. The traditional approach is to train a separate model for each budget, but this incurs enormous training and maintenance overhead. Looped Transformers support flexible inference compute through parameter reuse, but maintaining strong performance across multiple budgets remains an open problem.

The core idea of Accordion is: rather than adding model parameters, dynamically adjust the effective depth through fixed-prefix layer looping. Each budget corresponds to a predefined execution path, and the lowest budget retains the original unlooped model. In this way, a single model can cover inference budgets from low to high without retraining multiple models.

Mixture of Experts (MoE) is a sparsely activated model architecture: the model contains multiple parallel "expert" subnetworks (usually feed-forward layers), but each input token is routed to only a few of them for computation, with the remaining experts inactive. This allows the model's total parameter count to be far larger than the number of parameters actually activated per forward pass, expanding model capacity while controlling compute cost. The key component of MoE is the gating/routing network, which decides which experts to activate based on token representations and typically introduces a load-balancing loss to avoid uneven expert utilization. Accordion is built on the MoE architecture, meaning it retains low per-token compute overhead while scaling parameters — providing a good cost foundation for subsequently adjusting effective depth dynamically through layer looping.

A looped Transformer refers to repeating some layers (or all layers) multiple times during inference or training, thereby increasing effective computational depth without adding parameters. The intuition is that the same set of weights can be called repeatedly, effectively simulating a deeper network with fewer parameters. However, looped Transformers face several challenges: repeatedly executing the same layers can lead to representation degradation or training instability; performance varies greatly across different loop counts, making it difficult to cover multiple compute budgets with a single model; moreover, the extra computation from looping does not always translate linearly into performance gains, and diminishing returns exist. Accordion's fixed-prefix layer looping targets precisely these issues by restricting looping to prefix layers and combining it with budget-aware training to stabilize multi-budget performance.

Method: Fixed-Prefix Looping and Budget-Aware Training

Accordion is based on the Mixture-of-Experts architecture and supports five inference budgets. Each budget corresponds to a different number of loops, but looping is restricted to the model's prefix layers — hence "fixed-prefix layer looping." This design increases inference depth while controlling the growth of computation.

Training is divided into three stages. In the joint pre-training stage, each micro-batch randomly samples a budget and optimizes only its next-token prediction loss, without adding cross-budget consistency objectives. This avoids the complexity of multi-task training while enabling the model to learn to work under different budgets.

In the supervised fine-tuning stage, three domain experts are integrated into a single model through parameter merging and budget-matched multi-teacher on-policy distillation. The reinforcement learning stage further maintains performance across budgets. The final model surpasses the corresponding domain experts on the average performance of their respective tasks and budgets, while maintaining strong instruction-following ability across all budgets and preserving depth-related gains in mathematical reasoning and code generation.

On-policy distillation is a variant of knowledge distillation: the student model samples from its own generated output distribution, and the teacher model provides supervision signals (such as token-level probabilities or scores) on these samples, rather than merely imitating the teacher on fixed offline data. Because the student is corrected on trajectories it produces itself, it better mitigates the mismatch between training and inference distributions. In the supervised fine-tuning stage, Accordion adopts budget-matched multi-teacher on-policy distillation: for different inference budgets, it selects the domain expert that performs best under that budget as the teacher, and the student model learns from multiple teachers based on samples from its own policy, thereby integrating the capabilities of multiple domain experts into a single elastic model.

Parameter merging typically refers to fusing the weights of multiple independently trained models (or different fine-tuned versions of the same model) into a single model via averaging, task arithmetic, or Fisher weighting. Its advantage is that multiple expert capabilities can be combined without additional training, but simple averaging may degrade performance, so distillation or fine-tuning is often needed to recover. Accordion uses parameter merging to integrate three domain experts as initialization, then refines through budget-matched multi-teacher on-policy distillation so that the merged model approaches or surpasses the corresponding experts under each budget. This is also the key technical step behind its conclusion that "the final model surpasses domain experts on their respective tasks."

Experimental Results: Stable Gains from Increased Depth

After pre-training on 3.6T tokens, Accordion-6B-A0.5B's five-category macro average across 37 tasks improves by 3.01 percentage points as inference depth increases. The middle budget captures 89% of the gain using only half the additional layer executions, indicating that performance improvement does not depend linearly on computation and that an efficiency-optimal region exists.

The final model exhibits strong instruction-following performance across all budgets, and gains from increased depth persist in mathematical reasoning and code generation tasks. This means users can flexibly choose a budget based on task difficulty — saving computation on simple tasks and achieving better results on complex ones.

Significance and Outlook

Accordion offers a practical solution for elastic inference, covering multiple budgets with a single model and significantly reducing training and deployment costs. Its fixed-prefix looping design balances flexibility and efficiency, while multi-teacher distillation and parameter merging ensure performance across budgets. In the future, this approach is expected to extend to more budgets and broader task domains, advancing the deployment of large models in resource-constrained scenarios.

Paper link: Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping (example link, please replace with the actual link)

One-line Summary

Accordion enables a single MoE model to cover five inference budgets through fixed-prefix layer looping, improving average performance by 3.01 points across 37 tasks.

Paragraph Summary

The Accordion paper proposes an elastic-depth inference method based on Mixture of Experts (MoE) that uses fixed-prefix layer looping to let a single model support five inference budgets, avoiding the need to train a separate model for each budget. Training has three stages: joint pre-training randomly samples budgets and optimizes next-token loss; supervised fine-tuning integrates domain experts through parameter merging and budget-matched multi-teacher on-policy distillation; reinforcement learning further maintains performance across budgets. After 3.6T-token pre-training, Accordion-6B-A0.5B improves by an average of 3.01 percentage points across 37 tasks as depth increases, with the middle budget capturing 89% of the gain using only half the additional layer executions, while maintaining strong instruction-following ability across all budgets and preserving depth-related gains in mathematical reasoning and code generation.