Accordion
An elastic deep reasoning model based on a mixture-of-experts (MoE) architecture. Through a fixed-prefix layer looping mechanism, it supports multiple reasoning budgets without increasing the parameter count, enabling a single model to cover low to high computational budgets.
Timeline (last 90 days)
The paper "Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping" proposes Accordion, a mixture-of-experts (MoE)-based model that supports five inference budgets through fixed-prefix layer looping.
Accordion improves by an average of 3.01 percentage points across 37 tasks.
Accordion's intermediate budget captures 89% of the gains using only half the extra layer executions.
Accordion's core idea is not to add model parameters but to dynamically adjust effective depth through fixed-prefix layer looping.
Accordion's lowest budget retains the original model without looping.
Accordion's training proceeds in three stages: joint pre-training, supervised fine-tuning, and reinforcement learning.
In the joint pre-training stage, one budget is randomly sampled per micro-batch and only its next-token prediction loss is optimized, with no cross-budget consistency objective added.
In the supervised fine-tuning stage, Accordion integrates three domain experts into a single model through parameter merging and budget-matched multi-teacher on-policy distillation.
The final Accordion model surpasses the corresponding domain experts in average performance on their respective tasks and budgets.
Accordion exhibits strong instruction-following performance at all budgets, and the gains from increased depth persist in mathematical reasoning and code generation tasks.
All Facts (10)
The paper "Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping" proposes Accordion, a mixture-of-experts (MoE)-based model that supports five inference budgets through fixed-prefix layer looping.
50%UnverifiedAccordion improves by an average of 3.01 percentage points across 37 tasks.
50%UnverifiedAccordion's intermediate budget captures 89% of the gains using only half the extra layer executions.
50%UnverifiedAccordion's core idea is not to add model parameters but to dynamically adjust effective depth through fixed-prefix layer looping.
50%UnverifiedAccordion's lowest budget retains the original model without looping.
50%UnverifiedAccordion's training proceeds in three stages: joint pre-training, supervised fine-tuning, and reinforcement learning.
50%UnverifiedIn the joint pre-training stage, one budget is randomly sampled per micro-batch and only its next-token prediction loss is optimized, with no cross-budget consistency objective added.
50%UnverifiedIn the supervised fine-tuning stage, Accordion integrates three domain experts into a single model through parameter merging and budget-matched multi-teacher on-policy distillation.
50%UnverifiedThe final Accordion model surpasses the corresponding domain experts in average performance on their respective tasks and budgets.
50%UnverifiedAccordion exhibits strong instruction-following performance at all budgets, and the gains from increased depth persist in mathematical reasoning and code generation tasks.
50%