[KongchangAI]
· 3 min read· 1,553 words

Agent Self-Improvement with Frozen Weights: The Value and Limits of Failure Trajectories

Agent Self-Improvement with Frozen Weights: The Value and Limits of Failure Trajectories

Optimizing Agent notes with failure trajectories under frozen weights yields limited gains, mostly from search itself.

This study investigates whether AI Agent performance can be improved without updating model parameters, solely by optimizing the external harness. Using a 9B open-source model as executor and a larger teacher model to write notes, the team compared failure-trajectory-based notes against error-hidden notes on a customer service decision task. Failure trajectories yielded a 4.0pp gain, but ~60% was attributable to the search process itself. In full multi-turn conversations, no demonstrable improvement appeared. Strict screening passed only a few single experiences (unvalidated), and a neutral note scored higher than specific experiences. Blind review found 146 of 240 tool-use test items defective, highlighting evaluation quality concerns.

Does AI Agent self-improvement necessarily require updating model parameters? A recent study offers a different approach: keeping model weights frozen and improving Agent performance solely by optimizing the "harness"—the operational notes external to the model. The research team tested a 9-billion-parameter open-source model, with a larger teacher model writing notes for it, and focused on whether failure trajectories could provide additional gains. The conclusion: useful, but far less than it appears.

Agent self-improvement research source

Experimental Design: Frozen Weights, Notes Only

The core premise of the study is "Frozen Weights"—the Agent's model parameters are completely fixed, with no fine-tuning whatsoever. The only avenue for improvement is adjusting the harness, i.e., the prompts, rules, and operational instructions surrounding the model. The value of this design lies in its low cost, interpretability, and avoidance of catastrophic forgetting and deployment complexity associated with fine-tuning.

In the experiment, a 9-billion-parameter open-source model served as the executing Agent, while a larger teacher model was responsible for writing notes. On a generated customer service decision task, researchers compared two types of notes: one written based on the Agent's failure trajectories, and another with those errors hidden during writing. In both cases, the note writer was informed of the target rules.

Frozen Weights is a common strategy in transfer learning and lightweight adaptation: keeping pre-trained model parameters unchanged while only adding trainable or adjustable components externally. Compared to full fine-tuning, frozen weights offer the advantages of low computational cost, preservation of the original model's learned knowledge (avoiding catastrophic forgetting), and the ability to reuse the same model copy at deployment by simply switching external configurations. The trade-off is that the model's internal representations cannot be adjusted for new tasks—all improvements must be expressed through the input side (prompts, rules, tool interfaces). In recent years, "non-parametric adaptation" methods represented by prompt engineering, retrieval augmentation, and external memory are extensions of this approach.

Harness in the Agent context refers to the entire control layer wrapping the base model, including system prompts, tool invocation specifications, decision rules, output format constraints, and retry and fallback logic. It determines how model capabilities are organized into executable behavior sequences. Unlike model weights, the harness is a pure text or code-level configuration that takes effect immediately upon modification and can be audited line by line, thus naturally offering interpretability and low-cost iteration. "Optimizing the harness" in this article essentially means rewriting this control layer through operational notes generated by a teacher model, without touching model parameters.

Failure Trajectories Yield a 4-Point Gain, but Most Comes from Search Itself

Results showed that notes based on failure trajectories improved decision accuracy by 4.0 percentage points (compared to the control group with errors hidden). This number appears positive, but upon further decomposition, researchers found that the search process of "trying and retaining higher-scoring notes" alone contributed about 60% of the total improvement. In other words, the "net value" of failure trajectories is less than half.

This finding warrants caution. Many Agent improvement schemes may conflate the gains from the search process with the value of failure signals when reporting results. The study's controlled experimental design precisely separates the two, revealing that the marginal contribution of failure trajectories is far smaller than the surface numbers suggest.

Failure trajectories refer to the complete interaction records produced by an Agent when executing tasks that result in errors or low-quality outcomes, including inputs, intermediate reasoning steps, tool calls, and final outputs. Using them as learning signals stems from the "learning from mistakes" approach: compared to success cases, failure cases often more directly expose systematic biases in the model. However, failure trajectories also carry substantial noise unrelated to errors and require additional annotation or judgment mechanisms to confirm "what constitutes failure."

Separating the gains of the search process is the methodological key of this experiment. When researchers select the best-performing notes from multiple candidates, even if the candidate notes are completely random, as long as enough attempts are made and high scorers are retained, an increase in accuracy will be observed—this is the so-called "selection bias" or "winner's curse." To distinguish "the value of failure signals" from "the value of search itself," a control group must be set up that does not access failure trajectories but undergoes the same search and selection process. This article, through such a control design, found that about 60% of the improvement can be attributed to the search mechanism rather than the additional information carried by failure trajectories. This decomposition has cautionary significance for any scheme reporting "self-improvement" effects.

In Full Conversation Scenarios, the Gain Disappears

More critically, in complete customer service conversation scenarios, the same notes showed no demonstrable improvement. The difference between decision tasks and full conversations may lie in the fact that multi-turn interactions introduce more noise and state dependencies, making single-step decision improvements difficult to transfer directly to long-horizon conversations.

This highlights a real-world problem: single-point task improvements in the lab may not translate into gains in actual business scenarios. Agent self-improvement research needs evaluation methods that are closer to end-to-end.

"Single Experiences" Under Strict Screening: Few Survive, Not Yet Validated

The study also established a strict single-experience screening process. In the customer service domain, two experiences passed screening but remain to be tested on new data; in the tool-use domain, one experience was written without seeing the Agent's errors and showed no demonstrable gain on new tasks.

An interesting exploratory finding: a neutral note containing no operational procedures scored higher than the learned experiences. This may suggest that under frozen weights, overly specific experiences may introduce overfitting or misdirection, while concise neutral guidance preserves greater flexibility.

Blind Review Reveals Evaluation Noise: 146 of 240 Test Items Defective

The study concluded with a blind review, and the results were startling: of 240 tool-use test items, 146 were judged defective. This means the evaluation set itself has serious quality issues, with nearly 60% of test items unreliable. This finding raises fundamental questions about Agent evaluation methodology—if the test items themselves are defective, then conclusions of "improvement" or "no improvement" based on these tests may be distorted.

This also explains why some experimental results appear contradictory: under noisy evaluation, true signals are easily masked. The research team candidly flagged these limitations rather than concealing the problems, an attitude worth commending.

Blind review here refers to evaluators independently judging whether each test item is valid without knowing the item's source, expected answer, or experimental grouping. Its purpose is to eliminate evaluator bias from contaminating conclusions. When blind review finds that nearly 60% of test items are defective (e.g., ambiguous answers, erroneous premises, conflicts with target rules), it means the evaluation set itself lacks reliability. In measurement theory, this corresponds to a "construct validity" problem: the test items do not truly measure the capability they claim to measure.

The impact of evaluation noise on experimental conclusions is asymmetric: it can both mask genuine small improvements and be misread as effective due to occasional high scores. In Agent research with limited sample sizes and inherently small effect sizes, such noise is sufficient to flip conclusions between "improvement" and "no improvement." Therefore, when reporting gains, the defect rate of the evaluation set and the proportion of manual review should also be disclosed; otherwise, the reproducibility of the numbers is moot.

Implications for Agent Self-Improvement Practice

Overall, this study conveys several pragmatic messages. Failure trajectories as learning signals do have value, but under frozen weights, their independent contribution is limited and highly dependent on task form. The search process itself can deliver most of the gains, so engineering practice should prioritize ensuring the effectiveness of search and selection mechanisms.

Second, evaluation quality is the lifeline of Agent research. The blind review finding that nearly 60% of test items are defective reminds us that before pursuing "self-improvement," we must first ensure "reliable evaluation." Otherwise, all gain numbers may be castles in the air.

Finally, the exploratory finding that neutral notes outperform specific experiences may point in a counterintuitive direction: with frozen weights, less is more. Overly specific operational experiences may limit the model's generalization ability, while guidance that retains a certain level of abstraction is more robust. These conclusions still require more data validation but provide valuable hypotheses for future research.

One-line Summary

Optimizing Agent notes with failure trajectories under frozen weights yields limited gains, most of which come from the search itself.

Paragraph Summary

This study explores whether AI Agent performance can be improved without updating model parameters, solely by optimizing the external harness. The team used a 9-billion-parameter open-source model as the executor and a larger teacher model to write operational notes, comparing notes based on failure trajectories versus notes with errors hidden on a customer service decision task. Results showed that failure trajectories brought a 4.0 percentage point improvement, but about 60% could be attributed to the search and selection process itself. In complete multi-turn conversation scenarios, the same notes showed no demonstrable improvement. Under strict screening, only a few single experiences passed and remain unvalidated; a neutral note even scored higher than specific experiences. Blind review also found 146 of 240 tool-use test items defective, raising concerns about evaluation quality. The study suggests that the value of failure signals is limited, and that search mechanisms and evaluation reliability are the real keys.

#Agent self-improvement#frozen weights#failure trajectories#harness optimization#evaluation quality#AI Agent
Share:

Related articles