SPLIT
A planning method for optimal allocation of inference compute under imperfect verifiers, based on empirical Bayes fitting of a precise formula; it learns verifier behavior from a small amount of labeled pilot data and generalizes to new tasks.
Timeline (last 90 days)
SPLIT is a method for planning inference-time compute allocation using a small amount of labeled pilot data, used to decide, under a given budget, how many candidates to sample, which models to sample from, and how much compute to spend on verification.
SPLIT relies on an exact formula to compute the probability of submitting the correct answer under accept/reject and top-score selection strategies, a formula that accounts for variation in task difficulty, blind spots shared across models, and correlations among repeated verifier calls.
SPLIT fits its formula via empirical Bayes and uses the fitted results to plan budgets on unseen tasks.
The research team evaluated SPLIT on seven benchmarks, covering math, science QA, code, and software engineering agents, involving 26 models, 175 agent systems, and six real verifiers, all from public releases.
SPLIT's allocation scheme is only 0.02 to 0.8 percentage points away from the post-hoc optimal allocation.
SPLIT's formula explains a counterintuitive phenomenon: under certain conditions, adding more LLM judges actually reduces accuracy, because verifier errors are correlated, and calling the same or similar verifiers multiple times cannot independently eliminate errors and may instead amplify systematic bias.
When the verifier caps the achievable success rate, SPLIT outperforms power-law fitting as a predictor.
SPLIT's regret depends only on the quality of SPLIT's ranking of configurations, and SPLIT remains near-optimal under compute, price, and latency costs, as well as in online settings.
The research team reported scenarios where SPLIT fails.
All Facts (9)
SPLIT is a method for planning inference-time compute allocation using a small amount of labeled pilot data, used to decide, under a given budget, how many candidates to sample, which models to sample from, and how much compute to spend on verification.
50%UnverifiedSPLIT relies on an exact formula to compute the probability of submitting the correct answer under accept/reject and top-score selection strategies, a formula that accounts for variation in task difficulty, blind spots shared across models, and correlations among repeated verifier calls.
50%UnverifiedSPLIT fits its formula via empirical Bayes and uses the fitted results to plan budgets on unseen tasks.
50%UnverifiedThe research team evaluated SPLIT on seven benchmarks, covering math, science QA, code, and software engineering agents, involving 26 models, 175 agent systems, and six real verifiers, all from public releases.
50%UnverifiedSPLIT's allocation scheme is only 0.02 to 0.8 percentage points away from the post-hoc optimal allocation.
50%UnverifiedSPLIT's formula explains a counterintuitive phenomenon: under certain conditions, adding more LLM judges actually reduces accuracy, because verifier errors are correlated, and calling the same or similar verifiers multiple times cannot independently eliminate errors and may instead amplify systematic bias.
50%UnverifiedWhen the verifier caps the achievable success rate, SPLIT outperforms power-law fitting as a predictor.
50%UnverifiedSPLIT's regret depends only on the quality of SPLIT's ranking of configurations, and SPLIT remains near-optimal under compute, price, and latency costs, as well as in online settings.
50%UnverifiedThe research team reported scenarios where SPLIT fails.
50%