FOCUS: Active Critic Learning Solves Long-Horizon Agent Credit Assignment

Long-horizon language model agents face a core challenge under sparse outcome rewards: credit assignment. When an agent executes dozens or even hundreds of steps and receives only a final reward, how do you determine which step actually mattered? Traditional methods either rely on learned critic networks to estimate values of intermediate states, or repeatedly sample continuations from intermediate points in the trajectory to estimate value. But the former yields inaccurate value estimates, and the latter is costly. The recently proposed FOCUS method attempts to mitigate both problems simultaneously using "selective branching" and active learning, achieving clearly better performance than GRPO and VinePPO on multiple benchmarks.

The Credit Assignment Dilemma Under Sparse Rewards

Reinforcement learning for long-horizon agents is fundamentally different from single-step tasks. In mathematical reasoning or code generation, reward signals can usually be attributed relatively directly to the output. But in tasks like legal reasoning, deep search, and code localization, the agent must gradually collect information and adjust its strategy across multiple rounds of interaction before finally receiving a correct/incorrect judgment. How this final signal is propagated back to each earlier decision is the credit assignment problem.

According to the paper, sparse outcome rewards make it difficult for the agent to assign credit or blame to earlier actions. Learned critic networks can provide value estimates for intermediate states, but these estimates may be inaccurate; meanwhile, using repeated continuation sampling to estimate intermediate values incurs substantial computational overhead at multiple points along the trajectory. Both paths have their costs, and FOCUS's design philosophy is to find a more economical balance between the two.

Critic network is a value function approximator in actor-critic reinforcement learning architectures, typically sharing or having independent parameters from the policy network (actor), responsible for outputting a scalar value estimate for a given state (or state-action pair). In long-horizon agent scenarios, the critic needs to learn to judge "given that we've reached this step, how likely is subsequent success?" when only final outcome rewards are available. This is essentially a regression problem, with training signals coming from the returns of actual rollouts. The problem is that when trajectories are long and rewards are extremely sparse, the samples the critic sees early on are almost all failed trajectories, causing value estimates to become highly homogeneous or even collapse, leading to distorted advantage estimates. A direct motivation for FOCUS's selective branching is to provide the critic with more discriminative training signals by actively collecting successor outcomes for specific states.

Selective Branching: Spending Sampling Budget Where It Matters

FOCUS's core mechanism is "selective branching." Unlike traditional methods that uniformly sample continuations at every state, FOCUS's rollout controller actively selects those states that are "expected to most reduce the critic's estimation error" for continuation sampling. In other words, it no longer blindly samples everywhere, but lets the critic network actively "ask questions": On which states am I least confident? Then it gathers additional evidence for those states.

This active-learning-style sampling strategy brings two benefits. First, the sampling budget is concentrated in the regions where the critic is most uncertain, improving the information gain per unit of sampling cost. Second, the sampled evidence is used both to improve the critic's value estimates and as training targets to feed back into the critic network, forming a closed loop. The paper describes this as "helping the critic network learn its value estimator through selective branching."

VinePPO is one of FOCUS's main comparison baselines. Its core idea is to perform multiple continuation samples at intermediate states of the trajectory, using the mean empirical return of these continuations as a Monte Carlo estimate of that state's value, thereby replacing the learned critic. This approach bypasses the problem of inaccurate critic valuations, but the cost is that each intermediate point requires additional rollouts, with sampling overhead growing linearly with the number of branch points. FOCUS's selective branching can be seen as a "sparsification" of VinePPO: instead of doing continuations at every state, the controller selects the subset of states with the highest information content. Understanding this comparison helps clarify FOCUS's positioning—it does not abandon continuation sampling, but rather changes VinePPO's uniform budget into an adaptive budget.

TD(λ) Advantages and Modular Context Management

After obtaining more refined value estimates, FOCUS uses TD(λ) advantages to score segments between consecutive states. TD(λ) is a classic technique in reinforcement learning that strikes a balance between bias and variance, making it suitable for credit assignment in long-horizon scenarios. The refined value estimates combined with TD(λ) make segment-level advantage computation more reliable.

Another engineering point worth noting is the modular context management interface. The context of long-horizon agents keeps expanding, making compaction almost essential. FOCUS decouples context management from rollout allocation, so that segment boundaries produced by compaction can be directly used for value estimation and credit assignment. This design keeps the method flexible when facing different context management strategies, rather than tying the two together.

TD(λ) is a multi-step return method in temporal difference learning that uses a parameter λ∈[0,1] to perform exponential weighting interpolation between the single-step TD target (low variance, high bias) and the Monte Carlo return (high variance, low bias). When λ=0, it degenerates to single-step TD; when λ=1, it degenerates to the full trajectory return. In long-horizon credit assignment, TD(λ)'s advantage is that it can gradually propagate the final outcome reward back along the trajectory while using value estimates to truncate excessively long return chains to control variance. FOCUS uses TD(λ) to compute segment-level advantages after obtaining more refined critic valuations, which is equivalent to letting "trustworthy intermediate valuations" play the bootstrap role, decomposing the sparse terminal reward into a series of learnable local signals. The choice of λ directly affects the bias-variance tradeoff, and practitioners typically need to adjust it according to task length.

Experimental Results: Surpassing Strong Baselines on Three Benchmarks

The paper conducted experiments on three types of benchmarks—legal reasoning, deep search, and code localization—comparing against GRPO and VinePPO under matched per-step generation budgets. According to the paper's data, at the Qwen3.5-9B scale, FOCUS improved over the strongest baseline by 7.2 percentage points on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 percentage points on DeonticBench (56.5 vs. 52.4).

These improvement margins are not small in reinforcement learning comparisons, especially under matched budgets, indicating that the gains do not come from more sampling volume but from the choice of sampling locations and improvements in the credit assignment mechanism. However, it should also be noted that the experiments are concentrated at a specific scale (9B) and specific benchmarks, and the method's generalizability to larger models and broader tasks remains to be verified.

Implications for Long-Horizon Agent Training

FOCUS's approach reflects a broader trend: reinforcement learning for long-horizon agents is shifting from "piling on compute and sampling" to "allocating sampling more intelligently." As trajectory length increases, the cost of uniform sampling rises sharply, and actively selecting high-information states for branching may be a more sustainable path.

That said, the design of decoupling context management from rollout allocation is also worth learning from. Context compaction is unavoidable in long-horizon tasks, and if compaction boundaries can also serve credit assignment, that's killing two birds with one stone. For teams building agent training pipelines, the modular perspective FOCUS provides is more valuable as a reference than any single algorithmic detail.

Of course, active-learning-style critic training also introduces additional complexity: accurately estimating "which state can most reduce error" is itself a problem, and errors in the controller itself may affect sampling efficiency. The paper does not elaborate on this in the abstract, and practitioners need to assess the tuning costs themselves when implementing it in practice.

One-line Summary

FOCUS uses selective branching and active learning to improve credit assignment for long-horizon agents, surpassing GRPO and VinePPO on multiple benchmarks.

Paragraph Summary

Long-horizon language model agents face a credit assignment challenge under sparse outcome rewards: final rewards are difficult to trace back to early decisions. Traditional methods either rely on inaccurate critic networks or uniformly sample continuations at intermediate trajectory points, resulting in high costs. FOCUS proposes a selective branching mechanism in which a rollout controller actively selects states that can most reduce the critic's estimation error for continuation sampling, concentrating the sampling budget in high-information regions and feeding the sampled evidence back to the critic to form a closed loop. Combined with TD(λ) advantage computation and modular decoupled context management, FOCUS outperforms GRPO and VinePPO on legal reasoning, deep search, and code localization benchmarks at the Qwen3.5-9B scale, improving BrowseComp-Plus by 7.2 points and DeonticBench by 4.1 points under matched budgets. Its generalizability and the controller's own errors remain open questions.