Turn-Level Best-of-N Sampling: A New Strategy for Inference-Time Scaling of LLM Agents

Introduction: Why Do Agents Need Inference-Time Scaling?

Large language model (LLM) agents often fail on long-horizon tasks due to a single misstep in decision-making. Traditional methods rely on optimization during training, but inference-time scaling offers a lightweight alternative. A recent paper proposes Turn-Level Best-of-N Sampling, which improves agent performance on complex tasks through a generator-selector framework.

Inference-time scaling refers to improving output quality by increasing computation during the model's inference phase without modifying model parameters. Common forms include Chain-of-Thought, self-consistency sampling, and Best-of-N sampling. Compared with training-time scaling (such as increasing model size or training data), inference-time scaling is more flexible and cheaper to deploy, making it especially suitable for scenarios where frequent retraining is not feasible. For LLM agents, every decision in a long-horizon task can affect subsequent states, so investing extra computation at key decision points for candidate generation and selection is a natural scaling approach.

Method: How Does the Generator-Selector Framework Work?

The study uses two LLMs: a generator produces N candidate actions at each turn, and a selector picks one to execute. Unlike Zainullina et al. (2025), which uses a critic model for guided search, this paper evaluates whether off-the-shelf open-weight LLMs can directly compare candidate actions. This approach does not rely on additional training and achieves performance gains solely through inference-time computation.

Differences from Related Work

Prior work (such as Zainullina et al.) relies on a dedicated critic model for guided search, whereas this study explores the feasibility of using a general-purpose open-source LLM as the selector. This design lowers the deployment barrier because no additional reward model or critic model needs to be trained.

Best-of-N sampling is a classic inference-time scaling technique: generate N candidate outputs for the same input, then use some scoring mechanism to select the best one. In the agent setting, "turn-level" means performing sampling and selection at each decision turn (rather than at the end of the entire trajectory), thereby correcting single-step errors in a timely manner. In the generator-selector framework, the generator is responsible for producing diverse candidate actions, and the selector acts as the scoring function. The selector can be a reward model, a critic model, or a general-purpose LLM that makes comparative judgments via prompting. The key difference in this paper is the use of an off-the-shelf open-weight LLM to directly compare candidate actions, avoiding the overhead of training a dedicated selector.

Experimental Results: Improvements in Both Accuracy and Consistency

Experiments on long-horizon terminal tasks show that turn-level Best-of-N sampling delivers significant performance improvements: accuracy increases by 5-10%, and consistency across repeated attempts is also markedly enhanced. Analysis of agent trajectories reveals that agent behavior shifts toward ideal patterns, such as fewer attempts, better adherence to task instructions, and enhanced debugging and testing behavior.

Changes in Behavioral Patterns

Trajectory analysis shows that the agent no longer engages in frequent trial and error but instead solves problems more systematically. The increase in debugging and testing behavior indicates that the agent places greater emphasis on verification during task execution, which may be a key factor behind the accuracy gains.

Trajectory analysis is an important means of evaluating agent behavior quality, typically by examining the complete record of the agent's interaction with the environment (including action sequences, observations, and intermediate reasoning) to identify behavioral patterns. Common analysis dimensions include: number of attempts, degree of instruction following, debugging behavior (such as checking error logs and retrying failed commands), and testing behavior (such as running validation scripts). Trajectory analysis not only explains the source of performance improvements but also reveals where the agent is prone to errors, providing direction for future improvements. This paper compares trajectories before and after sampling and finds that agent behavior shifts toward more systematic patterns.

In-Depth Analysis: The Potential and Limitations of Inference-Time Scaling

The core advantage of this method is that it is simple and effective: performance can be improved through sampling and selection alone without additional training. However, the capability of the selector is crucial—if the selector cannot accurately compare the quality of actions, the effectiveness of the entire framework is diminished. In addition, computational cost rises as N increases, so a trade-off between performance and efficiency is needed.

Implications for Agent Design

Turn-level sampling refines inference-time scaling from the entire trajectory level to each decision point, which better matches the characteristics of long-horizon tasks. Future work could explore more efficient selector designs or combine this method with other inference-time scaling techniques.

Conclusion

Turn-level Best-of-N sampling provides a practical inference-time scaling solution for LLM agents, delivering measurable improvements in both accuracy and consistency. Its training-free, plug-and-play nature gives it high application value.

One-line Summary

Turn-level Best-of-N sampling improves LLM agent accuracy by 5-10% without training.

Paragraph Summary

This article introduces an inference-time scaling method called turn-level Best-of-N sampling for improving the performance of large language model agents on long-horizon tasks. The method uses a generator-selector framework: the generator produces N candidate actions at each turn, and the selector picks one to execute. Unlike prior work that relies on a dedicated critic model, this study evaluates the feasibility of using an off-the-shelf open-weight LLM as the selector without additional training. Experiments on long-horizon terminal tasks show that the method improves accuracy by 5-10% and markedly enhances consistency across repeated attempts. Trajectory analysis shows that agent behavior shifts toward ideal patterns, including fewer attempts, better instruction following, and enhanced debugging and testing behavior. The article also discusses limitations such as selector capability and computational cost, and notes that future work could explore more efficient selector designs or combinations with other inference-time scaling techniques.