The Mystery of Agent Performance Gains: Eliciting Potential or Expanding Boundaries?

Introduction: The Key Question Behind Agent Performance Gains

Recent research on agent harness optimization shows that the performance of language model agents can be significantly improved without updating model weights. Well-designed harnesses more effectively unlock the underlying model's capabilities by shaping context, tool use, and execution workflows. However, raw success rates alone cannot fully characterize whether an optimized harness succeeds more reliably within existing capability bounds (elicitation) or expands the range of solvable tasks (expansion). This distinction is critical for recursive self-improvement and evolutionary search, since expansion may provide broader successful experiences to push capability boundaries. Despite its growing importance, this distinction remains underexplored in harness optimization.

Agent harness: A runtime system built around a language model that handles task parsing, context management, tool invocation, execution flow control, and result feedback. It does not change model weights; instead, it guides model behavior through external designs such as prompt engineering, memory mechanisms, tool interfaces, and error recovery strategies. The goal of harness optimization is to more fully leverage the model's existing capabilities without retraining it. Common optimization approaches include improved task decomposition, additional reflection steps, and optimized tool selection logic. Because the harness and model capabilities are coupled, evaluating whether harness optimization "elicits" the model's existing potential or "expands" the range of tasks it can solve has become a key question for understanding the sources of progress in agent systems.

Methodology: A Framework Borrowed from RL Analysis

To systematically analyze the gains from harness optimization, researchers borrowed the pass@k analysis method used in reinforcement learning (RL) literature to compare models before and after training, comparing an optimized harness with its pre-optimization counterpart. The evaluation covers three aspects: changes in reachable task coverage, a comparison of harness optimization against model scaling and increased inference-time compute, and how gains vary with task difficulty. This multi-angle analysis aims to reveal the nature of harness optimization gains.

pass@k metric: In code generation and RL evaluation, pass@k means allowing the model to generate k samples, and success is counted if at least one of them passes the tests or correctly completes the task. Its core idea is to measure the model's potential capability ceiling across multiple attempts, rather than the reliability of a single attempt. When k=1, pass@1 reflects the first-attempt success rate and is often used to measure single-attempt performance in real deployment; as k increases, pass@k gradually approaches the model's "reachability" coverage on a given task. By comparing the pass@k curves of a harness before and after optimization, one can determine whether the performance gain comes from more stable single-attempt success (a vertical shift of the whole curve) or from expanding the boundary of solvable tasks (a marked rise at high k). This metric can effectively distinguish "elicitation" from "expansion" effects.

Core Findings: The Complexity of Gains and the Reversal Phenomenon

In static prediction and dynamic tasks, the research finds that the pass@1 advantage of a harness over stronger interventions (such as model scaling or increased inference compute) may reverse under larger attempt budgets. In other words, when more attempts are allowed, a harness that originally performed better in single attempts may no longer hold its advantage. Moreover, aggregate gains mask shortcomings on difficult tasks—harness optimization may yield significant improvements on easy tasks but may underperform on difficult ones. These findings highlight the need for fine-grained diagnostics.

pass@1 advantage reversal: This refers to the situation where, under a single attempt (pass@1), an optimized harness may outperform stronger interventions such as model scaling or increased inference compute; but when the attempt budget grows (e.g., pass@10, pass@100), the pass@k curves of stronger interventions may overtake harness optimization. This reversal suggests that the gains from harness optimization may be concentrated in improving single-attempt reliability rather than expanding the reachable task boundary. Conversely, model scaling or increased inference compute often covers a broader task space and shows greater potential under high attempt budgets. Therefore, comparing different interventions solely by pass@1 can be misleading; performance under multiple attempt budgets should be considered for a comprehensive judgment.

Implications and Outlook: Toward a More Fine-Grained Evaluation System

This research reminds us that the gains from agent harness optimization are not one-dimensional. Measuring them simply by success rate improvement may obscure changes in the capability distribution. In the future, evaluating harness optimization requires distinguishing elicitation from expansion effects and making comprehensive judgments that account for factors such as task difficulty and attempt budget. This cannot be ignored for building more reliable agent systems and advancing recursive self-improvement.

One-line Summary

Harness optimization gains require distinguishing elicitation from expansion; pass@1 advantages may reverse as attempt budget increases.

Paragraph Summary

This study borrows the pass@k analysis method from reinforcement learning to systematically compare agent harness performance before and after optimization, aiming to distinguish whether gains come from elicitation (succeeding more reliably within existing capabilities) or expansion (expanding the range of solvable tasks). The research finds that the pass@1 advantage of harness optimization over stronger interventions such as model scaling or increased inference compute may reverse under larger attempt budgets; meanwhile, aggregate success rate improvements may mask shortcomings on difficult tasks. This indicates that measuring harness optimization solely by success rate obscures changes in the capability distribution, and future evaluation should perform fine-grained diagnostics combining task difficulty and attempt budget.