Sampling, Mixing, or Verifying? Optimal Allocation of Inference Compute Under Imperfect Verifiers

Title

Sampling, Mixing, or Verifying? Optimal Allocation of Inference Compute Under Imperfect Verifiers

Meta Description

SPLIT uses a small amount of labeled pilot data to precisely plan inference-time compute allocation, approaching hindsight-optimal on seven benchmarks and 26 models, and reveals conditions under which more LLM reviewers can reduce accuracy.

Article Content

Inference-time compute scaling has become a common means to improve the accuracy of language models and agents: generate many candidate answers, then let a verifier pick the best one. However, verifiers in real-world settings—unit tests, majority voting, LLM reviewers, reward models—are not perfect; they accept wrong answers, reject correct ones, and even make correlated errors. Under a fixed budget, developers must decide how many candidates to sample, from which models to sample, and how much compute to spend on verification. Today these decisions mostly rely on rules of thumb. A new paper proposes the SPLIT method, which attempts to answer these questions systematically using a small amount of labeled pilot data.

The Compute Allocation Dilemma Under Imperfect Verifiers

A common way to make language models more accurate is to increase inference-time compute: generate many candidate solutions and submit the one the verifier prefers. Success requires two conditions to hold simultaneously: some candidate is correct, and the verifier selects it. The first condition grows predictably with the number of samples; the second does not. Real verifiers accept wrong answers, reject correct ones, and their errors are often correlated.

Under a fixed budget, developers face three decisions: how many candidates to sample, from which models to sample, and how much compute to spend on verification. Currently these choices are usually made by rules of thumb, lacking theoretical guidance. The core contribution of SPLIT is to turn these decisions into an optimizable problem.

SPLIT's Core Mechanism: From Pilot Data to Exact Formulas

SPLIT relies on an exact formula that computes the probability of submitting a correct answer under accept/reject and top-score selection strategies. The formula accounts for variations in task difficulty, blind spots shared across models, and correlation between repeated verifier calls. SPLIT fits this formula via empirical Bayes and uses it to plan budgets on unseen tasks.

The key advantage of this approach is that it does not require large-scale trial and error for each new task; instead, it learns the verifier's behavior patterns from a small amount of labeled pilot data and then generalizes to new tasks.

Empirical Bayes is a statistical estimation method: it does not assume the prior distribution is completely unknown, nor that the prior is completely known; instead, it estimates the hyperparameters of the prior from data. In the context of SPLIT, this means researchers do not need to presuppose the verifier's accuracy or error correlation structure; instead, they use a small number of labeled pilot tasks (model-generated candidate answers and verifier judgments) to fit the distribution of these parameters, and then apply the fitted prior to new tasks for budget planning. Compared with redoing large-scale trial and error for each new task, empirical Bayes can significantly reduce estimation variance when data is limited, and it is the key statistical foundation that enables SPLIT to turn "a small amount of labels" into "generalizable planning."

Hindsight-optimal allocation refers to the theoretically optimal budget configuration chosen by looking back after all outcomes are known. It is equivalent to a "god's-eye view" baseline: the upper bound on accuracy that developers could have achieved if they had known in advance which configuration was best. Since outcomes cannot be known in advance in real settings, any planning method can only approach this upper bound, and the gap between the two is the "regret." The 0.02 to 0.8 percentage point gap reported by SPLIT is relative to this unattainable baseline, so this metric measures how close the planning method is to the theoretical limit, not the absolute level of accuracy.

Experimental Validation: Large-Scale Evaluation Across Seven Benchmarks and 26 Models

The research team evaluated SPLIT on seven benchmarks, covering math, science QA, code, and software engineering agents. The experiments involved 26 models, 175 agent systems, and six real verifiers, all from public releases.

The results show that SPLIT's allocation scheme differs from the hindsight-optimal allocation by only 0.02 to 0.8 percentage points. In contrast, all tested fixed rules lose at least 2.5 percentage points in some settings. This gap remains consistent across multiple domains and model scales, indicating that SPLIT's planning capability is robust.

The paper also points out that regret depends only on the quality of SPLIT's ranking of configurations, and that SPLIT remains near-optimal under compute, price, and latency costs, as well as in online settings.

An Additional Finding from the Formula: When More LLM Reviewers Reduce Accuracy

SPLIT's formula also explains a counterintuitive phenomenon: under certain conditions, adding more LLM reviewers actually reduces accuracy. This is because verifier errors are correlated; calling the same verifier or similar verifiers multiple times does not independently eliminate errors, and may instead amplify systematic bias.

This finding has direct implications for practitioners: blindly increasing the number of verifiers is not always effective; planning must be based on the verifier's error structure.

Correlation of verifier errors is central to understanding this counterintuitive phenomenon. If the errors of multiple reviews were independent of one another, then increasing the number of reviews would, like multiple independent votes, dilute random errors according to the law of large numbers and improve the reliability of the overall judgment. But in reality, the same LLM reviewer often shares the same training biases, prompt-understanding patterns, and knowledge blind spots across different calls, and similar models also share common systematic weaknesses. This correlation means that the "independent information" brought by additional reviews is far less than the nominal count, and may even repeatedly reinforce the same type of error, causing majority voting or weighted aggregation to be dominated by systematic bias. SPLIT's formula explicitly models this correlation structure, thereby making it possible to determine under what conditions adding reviews yields a net gain and under what conditions it yields a net loss.

As a Predictor: Beyond the Boundaries of Power-Law Fitting

When the verifier caps the ceiling on success, SPLIT outperforms power-law fitting as a predictor. The research team also reports scenarios where SPLIT fails, pointing the way for future research.

Overall, SPLIT pushes inference compute allocation from rules of thumb toward data-driven planning, providing a new tool for building more reliable AI systems.

Power-law scaling is an empirical extrapolation tool commonly used in inference-time scaling research: it assumes that accuracy rises smoothly as a power function with increases in the number of samples or compute, and fits a curve using a small number of data points to predict performance under larger budgets. This method performs well in scenarios where "sampling more will always find the correct answer," but it implicitly assumes that the probability of success will continue to approach 1. When the verifier itself has inherent flaws—accepting wrong answers and rejecting correct ones—the probability of success is capped by verifier quality, and the power-law curve overestimates the actual gain. By explicitly modeling verifier behavior, SPLIT can give more realistic predictions when such a success ceiling exists, which is why it outperforms power-law fitting on prediction tasks.

One-line Summary

SPLIT uses a small amount of labeled pilot data to plan sampling and verification budgets, approaching hindsight-optimal.

Paragraph Summary

Inference-time compute scaling—generating many candidates and letting a verifier pick—has become a common means to improve language model accuracy, but real verifiers accept wrong answers, reject correct ones, and their errors are correlated, so decisions about how many to sample, from which models, and how much compute to spend on verification have long relied on rules of thumb. The new paper's SPLIT method uses an exact formula to characterize the probability of submitting a correct answer under accept/reject and top-score selection strategies, and uses empirical Bayes to fit verifier behavior from a small amount of labeled pilot data and generalize to unseen tasks for budget planning. Evaluations on seven benchmarks, 26 models, 175 agents, and six real verifiers show that SPLIT differs from hindsight-optimal allocation by only 0.02 to 0.8 percentage points, while fixed rules lose at least 2.5 percentage points in some settings. The formula also explains the counterintuitive phenomenon that adding more LLM reviewers can reduce accuracy, and it outperforms power-law fitting when the verifier caps the success ceiling.