ABACUS Framework: Program-Guided Frozen Judgment Models for Multi-Step Reasoning

Title

ABACUS Framework: Program-Guided Frozen Judgment Models for Multi-Step Reasoning

Meta Description

By composing the decisions of a frozen judgment model with reusable programs, the ABACUS framework achieves 99.67% accuracy on BIG-Bench Hard tasks—35 percentage points above using Kev directly—along with 8.43× online speedup.

Article Content

Judgment models excel at semantic scoring over a limited set of candidate outcomes, but how can their local decisions be organized into powerful multi-step reasoning? The ABACUS framework offers a new approach: compose the decisions of a frozen judgment model through reusable programs, substantially improving reasoning capability while keeping model parameters unchanged.

Core Problem: The Reasoning Bottleneck of Judgment Models

Conventional judgment models typically output a final answer directly, skipping intermediate reasoning steps. On complex tasks such as BIG-Bench Hard, this "one-step" approach yields limited accuracy. ABACUS starts from a straightforward premise: since judgment models can reliably answer local semantic questions, can we design a set of programs that chain these local answers together into a multi-step reasoning chain?

The key to the framework is "frozen": the judgment model's parameters are never updated, and all reasoning capability comes from the compositional design of external programs. This avoids fine-tuning costs and makes the reasoning process more interpretable and reusable.

BIG-Bench Hard is a subset of 23 high-difficulty tasks selected from the BIG-Bench benchmark, covering categories such as multi-step arithmetic, logical reasoning, causal judgment, and date understanding. Its defining feature is that even large language models perform significantly below the human average on it, which is why it is often used as a touchstone for measuring multi-step reasoning ability. Unlike conventional question-answering benchmarks, correct answers on BBH tasks often depend on correctly chaining several intermediate steps, and single-step intuitive answers are prone to error—precisely the scenario where the judgment model's "one-step" strategy exposes its weaknesses. ABACUS selects three of these tasks for validation, aiming to test the gains from programmatic composition on tasks with longer reasoning chains and decomposable local semantic judgments.

Framework Design: A Teacher Generates Programs, the Judgment Model Executes Them

ABACUS uses a generative teacher to design judgment patterns and composition programs offline, while the judgment model solves local semantic questions online and the saved programs combine decisions into final answers. The research explores three ways of allocating design effort between humans and the teacher, with a focus on building through a shared interface.

In terms of implementation, the team instantiated the framework with Kev. The teacher model plans reasoning steps and generates program code, while the frozen judgment model acts as a "semantic executor," scoring candidate answers at each step. This division of labor makes program reusability possible—the same set of programs can be transferred to new inputs.

Kev here refers to a reasoning framework/interface based on judgment models that allows a frozen model to score candidate answers via natural-language or structured instructions, rather than relying on the model to directly generate a complete response. Using Kev as the instantiation vehicle means that ABACUS's programs do not require the model to have code-generation or tool-calling capabilities; the programs themselves handle control flow and composition logic, while Kev is only responsible for providing local semantic scores at each step. This decoupling allows the same set of programs to be reused across different inputs and makes it easier to pinpoint which step's judgment an error occurred at. Understanding Kev's role helps clarify the boundary of the division of labor in "the teacher designs programs, the judgment model executes them": the programs handle "how to think," while the model handles "how to judge."

Experimental Results: A Win-Win in Accuracy and Efficiency

On the three selected BIG-Bench Hard tasks, nine independently constructed programs solved 450 executions over 150 development inputs. The deployment study froze three programs refined through reference-consistency on the development set and evaluated them on 600 held-out questions.

The results are striking: the programs achieved 99.67% macro-averaged accuracy, compared with only 64.00% for using Kev directly and 68.80% for Qwen3.5-4B outputting only the final answer. Compared with the 99.07% observed accuracy of Qwen's unrestricted response strategy, ABACUS maintains a similar level while achieving an 8.43× online speedup—0.255 seconds versus 2.150 seconds per question.

This means that a compact judgment model paired with carefully designed programs can approach the reasoning performance of a much larger model without adding parameters, and with a clear speed advantage.

Macro-averaged accuracy is a metric computed by first calculating the accuracy for each task separately and then taking the arithmetic mean across tasks. Unlike micro-averaging (which pools all samples and computes overall accuracy), it gives each task equal weight, preventing tasks with larger sample sizes from dominating the overall score. In the evaluation across three BBH tasks, macro-averaging more evenly reflects the programs' performance across different reasoning types and prevents a high score on one task from masking failure on another. This is also why reporting per-task results alongside is the more prudent approach—a macro-average of 99.67% indicates that the three tasks are collectively near saturation, but the variance between tasks still needs to be judged in light of the per-task data.

Implications and Limitations

The value of ABACUS lies in demonstrating a reasoning-enhancement path of "programs + frozen models." It reduces multi-step reasoning's dependence on model scale and provides an inspectable intermediate structure for the reasoning process. However, the current experiments are concentrated on three specific tasks, and the generality of the program design and its cross-task transferability still require further validation. In addition, the offline design cost of the teacher model and the programs' sensitivity to task distribution are factors that need to be weighed in practical deployment.

Overall, this research opens up new space for the application of judgment models: rather than pursuing larger models, let programs become the "exoskeleton" of reasoning.

One-line Summary

ABACUS composes the local scores of a frozen judgment model with reusable teacher-designed programs to achieve high-accuracy multi-step reasoning.

Paragraph Summary

Targeting the problem that judgment models excel at local semantic scoring but lack the ability to organize multi-step reasoning, the ABACUS framework proposes a division of labor in which a generative teacher designs judgment patterns and composition programs offline while a frozen judgment model performs local scoring online. The research instantiates the framework with Kev on three BIG-Bench Hard tasks, where nine independently constructed programs completed 450 executions over 150 development inputs; in the deployment phase, three programs refined for consistency on the development set were frozen and achieved 99.67% macro-averaged accuracy on 600 held-out questions, significantly higher than the 64.00% of using Kev directly and the 68.80% of Qwen3.5-4B outputting only the final answer. Compared with the 99.07% of Qwen's unrestricted response strategy, accuracy is similar, but online speed is 8.43× faster (0.255 seconds versus 2.150 seconds per question). The authors believe this path reduces dependence on model scale and improves interpretability, but task coverage, program generality, and the teacher's offline design cost still require further validation.