ABACUS
A framework that performs multi-step reasoning by composing reusable programs over the local decisions of a frozen judge model. Model parameters are never updated throughout, and reasoning capability derives from the external program composition design.
Timeline (last 90 days)
The ABACUS framework improves multi-step reasoning by composing reusable programs to freeze the judgment model's decisions, and the model parameters are not updated throughout.
ABACUS uses a generative teacher to design judgment patterns and composition programs offline, while the judgment model solves local semantic problems online, and the saved programs compose decisions into the final answer.
The ABACUS team instantiated the framework with Kev: the teacher model plans reasoning steps and generates program code, and the frozen judgment model acts as a semantic executor that scores candidate answers for each step.
On three selected BIG-Bench Hard tasks, nine independently constructed programs solved 450 executions of 150 development inputs.
The deployment study froze three programs refined via reference-consistency on the development set and evaluated them on 600 held-out questions.
ABACUS programs achieve 99.67% macro-average accuracy on the three BIG-Bench Hard tasks.
ABACUS achieves an 8.43x online speedup, 0.255 seconds versus 2.150 seconds per question, while maintaining accuracy comparable to Qwen's unrestricted response strategy.
ABACUS's current experiments are concentrated on three specific tasks, and the generality of program design and cross-task transfer capability still require further validation.
The offline design cost of the teacher model and the programs' sensitivity to task distribution are factors that need to be weighed in ABACUS's practical deployment.
All Facts (9)
The ABACUS framework improves multi-step reasoning by composing reusable programs to freeze the judgment model's decisions, and the model parameters are not updated throughout.
50%UnverifiedABACUS uses a generative teacher to design judgment patterns and composition programs offline, while the judgment model solves local semantic problems online, and the saved programs compose decisions into the final answer.
50%UnverifiedThe ABACUS team instantiated the framework with Kev: the teacher model plans reasoning steps and generates program code, and the frozen judgment model acts as a semantic executor that scores candidate answers for each step.
50%UnverifiedABACUS's current experiments are concentrated on three specific tasks, and the generality of program design and cross-task transfer capability still require further validation.
50%UnverifiedThe offline design cost of the teacher model and the programs' sensitivity to task distribution are factors that need to be weighed in ABACUS's practical deployment.
50%UnverifiedABACUS programs achieve 99.67% macro-average accuracy on the three BIG-Bench Hard tasks.
50%UnverifiedABACUS achieves an 8.43x online speedup, 0.255 seconds versus 2.150 seconds per question, while maintaining accuracy comparable to Qwen's unrestricted response strategy.
50%UnverifiedOn three selected BIG-Bench Hard tasks, nine independently constructed programs solved 450 executions of 150 development inputs.
50%UnverifiedThe deployment study froze three programs refined via reference-consistency on the development set and evaluated them on 600 held-out questions.
50%