LLM Test-Time Ensembling: A New Theory of Two Sample Sizes and Weighted Ensembling

Test-time scaling has become an important path to improving large language model performance. By generating multiple answers and aggregating them, models can achieve significant gains without increasing parameter count. Weighted ensembling goes further by mixing answers from multiple models according to weights, where the weights determine how frequently each model is used. However, learning these weights from training data involves a subtle trade-off between two sample sizes—the number of labeled questions $Q$ and the number of generations $N$ per model per question. A recent theoretical study provides an in-depth analysis of their interaction, offering a new theoretical tool for understanding the learning limits of test-time ensembling.

Core Question: How Do the Two Sample Sizes Affect Ensemble Learning?

The weight learning process of weighted ensembling relies on two types of data resources: first, the number of labeled questions $Q$; second, the number of answers each model generates for each question $N$. Intuitively, more labeled questions allow the model to more accurately estimate the relative strengths of different models; more generations allow more reliable estimation of the details of each model's answer distribution.

The study focuses on the "clean risk," i.e., the population error of the ensemble under an infinite-generation voting rule. Under a margin condition with exponent $\beta$, the theoretical analysis shows that buffered empirical risk minimization (buffered ERM) with a calibrated margin buffer achieves excess risk on the order of $Q^{-1/2}+N^{-\beta/2}$ (ignoring logarithmic factors). This means that the number of questions and the number of generations affect ensemble performance at different power-law rates, and their contributions can be approximately decoupled.

Margin Condition and Fast Convergence

The margin condition characterizes how clear the "boundary" is between correct and incorrect model predictions. In classical statistical learning theory, if the proportion of samples near the true decision boundary is very low (i.e., the margin is large), then generalization error can decrease faster than the usual $Q^{-1/2}$ rate. A margin condition with exponent $\beta$ quantitatively describes this decay behavior: the larger $\beta$ is, the "cleaner" the boundary and the faster learning proceeds.

In the ensembling setting, the role of the margin condition is reflected in each model's distribution of answers to each question. When $\beta>1$, the $N^{-\beta/2}$ term in the excess risk may decay faster than $N^{-1/2}$, meaning that generations can yield more significant benefits when the boundary is clear. Conversely, if model outputs are very noisy ($\beta$ close to 0), increasing the number of generations does little to improve weight estimation. This also explains why the optimal test-time scaling strategy diverges across different models and tasks.

Three Learning Regimes: From Question Scarcity to Identification Limits

To investigate the optimality of this dependence, the researchers analyzed a simplified submodel: only two models, and only two possible answers per question. Using the Hellinger modulus for binomial mixture experiments, the minimax excess risk of this submodel is precisely characterized up to constants. Polynomial approximation and spectral analysis reveal three distinct regimes.

Regime 1: When questions are scarce, $Q$ dominates learning. When labeled questions are relatively scarce, the $Q^{-1/2}$ term dominates the excess risk, and generic buffered ERM is optimal. At this point, the marginal benefit of increasing the number of generations $N$ is limited, because the bottleneck in weight estimation is insufficient information at the question level.

Regime 2: In the intermediate region, cross-question pooling brings gains. In the intermediate region, pooling generation counts across questions is more effective than estimating each answer distribution separately, and only logarithmic gains can be obtained from additional questions. In other words, exploiting the cross-question structure of generation counts allows more efficient use of limited data.

Regime 3: When questions are sufficient, an identification floor appears. When the number of questions is sufficiently large, the minimax excess risk hits an identification floor of $N^{-\beta}$, which cannot be overcome even with infinitely many questions. This lower bound reflects the identifiability limit of the model's answer distribution itself and is independent of the number of questions.

Hellinger Modulus and Minimax Risk

The Hellinger modulus is a distance measure between probability distributions that integrates the difference between the square roots of the distribution densities. Compared with KL divergence, the Hellinger modulus has two advantages: first, it is bounded and does not diverge when distributions differ too much; second, it often yields tighter constant bounds in minimax lower-bound derivations for nonparametric estimation.

In the two-model binomial mixture submodel, the researchers use the Hellinger modulus to precisely characterize the distinguishability of the two mixture distributions. When the two models' accuracy rates are very close, the Hellinger distance approaches zero, meaning that with finite samples it is nearly impossible to distinguish which model is better—this is precisely the statistical root of the identification lower bound $N^{-\beta}$. The minimax framework then guarantees that this lower bound does not depend on the specific estimation method, reflecting the intrinsic difficulty of the problem itself. Polynomial approximation and spectral analysis further reveal the scaling behavior of the Hellinger distance with respect to $Q$ and $N$ under different regimes, thereby delineating the three learning regimes.

Statistical Intuition for Cross-Question Pooling

Pooling means not estimating the answer distribution for each question separately, but instead combining generation counts across all questions to estimate a shared weight parameter. Statistically, this is equivalent to estimating fewer parameters with more data, thereby reducing variance.

In the intermediate region, when estimating model strengths separately for each question, each question contributes only $N$ generations, resulting in high noise; after pooling, the total sample size reaches $Q \times N$, greatly improving estimation accuracy. However, pooling is not always optimal: when model strengths differ greatly across questions, forcing shared weights introduces bias. Theoretical research shows that pooling in the intermediate region yields only logarithmic gains, because heterogeneity across questions limits the benefit of the shared structure. This finding aligns with the "shared vs. specialized" trade-off in transfer learning and also provides theoretical clues for whether to model subgroups separately in practice.

Theoretical Significance and Practical Implications

The lower bound for the submodel also applies to general ensembles, matching the upper bound of buffered ERM (ignoring logarithmic factors) in part of the question-scarce and intermediate regions. This provides a nearly tight theoretical guarantee for weight learning in weighted ensembling.

From a practical perspective, the theory suggests that in test-time ensembling, strategy should be adjusted according to the number of available labeled questions. When questions are scarce, priority should be given to expanding the question set rather than blindly increasing the number of generations; when questions are abundant, improvements from more generations will encounter an identification ceiling, and simply increasing sampling will be difficult to sustain continued improvement in ensemble performance.

The paper also candidly points out open questions: the precise transition boundaries between different regimes remain unclear, and the construction of matching pooled estimators for general ensembles remains to be solved. These remaining issues point the way for future research.

Overall, this work places weight learning for test-time ensembling into a clear statistical learning framework, reveals the trade-off between $Q$ and $N$, and has important reference value for understanding and improving LLM ensembling strategies.

One-line Summary

A theory reveals the trade-off between the number of labeled questions and the number of generations in LLM test-time ensembling, delineating three learning regimes.

Paragraph Summary

This article interprets a theoretical study on weight learning for LLM test-time ensembling. The study focuses on two sample sizes—the number of labeled questions Q and the number of generations per question N—and proves that under a margin condition, buffered ERM achieves excess risk of Q^{-1/2}+N^{-β/2}. Through minimax analysis of a two-model submodel, it reveals three regimes: when questions are scarce, Q dominates; in the intermediate region, cross-question pooling yields only logarithmic gains; when questions are abundant, an N^{-β} identification floor appears that cannot be overcome even with infinitely many questions. The submodel lower bound can be generalized to general ensembles, providing a nearly tight theoretical guarantee for weight learning. Practically, it suggests adjusting strategy according to the number of questions and identifies open problems such as regime transition boundaries.