What Does Multi-Turn Reinforcement Learning Actually Improve in LLM Agents?

Multi-turn agentic reinforcement learning (Agentic RL) performs impressively on long-horizon tasks, but a new study reveals an often-overlooked fact: its performance gains depend heavily on interaction constraints. As the sampling width K and interaction depth T vary, the advantages brought by RL fluctuate significantly. This study systematically evaluates the real benefits of multi-turn RL through the Pass@(K,T) metric and proposes an alternative called BudgetMix RL in an attempt to mitigate the risk of budget dependence during deployment.

The Gains of Multi-Turn RL Are Not Unconditional

The research team's core finding is that the advantages of trajectory-level RL only emerge given a sufficient turn budget. Under the Pass@(K,T) framework, when the interaction depth T or the sampling width K tightens, the performance gains from RL noticeably decay. This means that many gains reported under generous budgets may be substantially diminished in real deployment environments.

This conclusion has direct implications for agent deployment. In practical applications, the number of API calls, context length, and response latency all constitute hard constraints, and it is difficult for models to obtain the ample interaction turns available during training. If the RL policy relies excessively on "trying a few more times" to solve problems, its performance may rapidly degrade when the budget is limited.

The Pass@(K,T) metric draws on the idea of Pass@K commonly used in code generation evaluation, but extends it to multi-turn interaction scenarios: K represents the number of trajectories independently sampled for the same task (sampling width), and T represents the maximum number of interaction turns allowed for a single trajectory (interaction depth). A task is only counted as passed if at least one of the K trajectories successfully completes it within T turns. The value of this metric lies in decoupling "model capability" from "consumable budget"—fixing K and T fixes the inference-time computation and call costs, thereby characterizing the true success rate of a model under a specific deployment budget. Compared with single-point Pass@1, the slope of the Pass@(K,T) curve as K and T vary directly reflects the sensitivity of performance to budget, making it a key tool for judging whether RL gains are "deployable."

Single-Step Decision Quality Is Not Uniformly Improved

A counterintuitive finding is that agentic RL does not uniformly improve the quality of every decision step. The research shows that its advantages are concentrated mainly in error recovery and long-horizon resilience: the model becomes better at adjusting its strategy after making a mistake and maintaining goal consistency in long-chain tasks, rather than becoming smarter at individual decision points.

This redefines our understanding of what RL does. It is more like training a kind of "resilience to setbacks" than directly injecting stronger reasoning ability. For developers, this means that when evaluating an agent, the focus should be on trajectories of recovery after failure rather than solely on single-step accuracy.

The distinction between trajectory-level RL and single-step decisions deserves clarification. Trajectory-level RL uses the final outcome of an entire multi-turn interaction (success/failure or cumulative return) as the learning signal, assigning credit to each action along the way through methods such as policy gradients; single-step supervision or process rewards, by contrast, directly score each intermediate decision. The former has sparse signals but aligns with the task objective, while the latter has dense signals but depends on high-quality process annotations. This explains why trajectory-level RL more easily shapes "recovery after mistakes and goal maintenance over long chains" behaviors, yet struggles to make the model more accurate at isolated decision points—because the sparse terminal reward mainly distinguishes the success/failure patterns of entire trajectories rather than fine-grained per-step merits.

Process Supervision Signals and BudgetMix RL

To address the deployment risk posed by budget dependence, the paper preliminarily explores the pros and cons of process supervision signals and proposes BudgetMix RL. This method actively varies the interaction budget during training, allowing the model to adapt to different execution constraints and thereby maintain robust performance across diverse deployment settings.

Experiments show that BudgetMix RL exhibits better robustness across a variety of execution settings. The practical value of this approach lies in the fact that, rather than pursuing peak performance under an ideal budget, it is better to train an agent that is insensitive to budget changes—something far more friendly to real product environments.

Process supervision refers to providing feedback signals on the intermediate steps of reasoning or interaction, rather than judging only the final result. In agent scenarios, it can take the form of immediate scoring or rewards for whether each tool call is reasonable and whether subgoals are achieved. Its advantages are providing denser learning signals and alleviating the credit assignment problem in long-horizon tasks; its drawbacks are high annotation costs, the possibility that the process rewards themselves are inaccurate, and the tendency for models to overfit to intermediate steps that "look correct" while deviating from the final objective. The paper's preliminary exploration of its pros and cons is precisely to show that, in budget-constrained multi-turn settings, relying solely on process signals is insufficient to solve the robustness problem—it also requires adjustments to training strategies at the budget level.

Implications for Agent Development

This research reminds us that when evaluating multi-turn RL agents, we cannot look only at scores under a single budget. Development teams should introduce multi-dimensional evaluation similar to Pass@(K,T) to clarify the performance boundaries of models under different interaction constraints. At the same time, budget randomization can be considered in training strategies to improve deployment adaptability.

The progress of agentic RL is real, but its boundaries are equally real. Understanding these boundaries is more meaningful than blindly chasing leaderboard improvements.

One-line Summary

The gains of multi-turn agentic RL depend on the interaction budget, and BudgetMix RL can mitigate the risk of budget dependence.

Paragraph Summary

This study points out that the performance gains of multi-turn agentic reinforcement learning are not unconditional but depend heavily on the interaction budget formed by sampling width K and interaction depth T. Systematic evaluation using the Pass@(K,T) metric reveals that when the budget tightens, the improvements brought by RL decay significantly, meaning that gains reported under generous budgets may be substantially diminished in real deployment. The study also finds that Agentic RL mainly improves error recovery and long-horizon resilience rather than uniformly improving single-step decision quality. To address the risk of budget dependence, the authors preliminarily explore the pros and cons of process supervision signals and propose BudgetMix RL—actively varying the interaction budget during training so that the model adapts to diverse execution constraints and performs more robustly across various settings.