Optimization Pitfalls in Self-Evolving LLM Agents: Value Concentration and Budget-Splitting Problems

Introduction: Wrapping Frozen Large Models in a "Harness"

In recent years, a growing approach has emerged for improving LLM agent performance without touching model weights — instead optimizing the "harness" that surrounds the model. A harness refers to the text scaffolding wrapped around the model, including persona definitions, task strategies, format rules, and control heuristics. The concept draws from software engineering's test harness idea — controlling and enhancing system behavior through external wrapping without modifying the core system. As models like GPT-4 and LLaMA have grown to billions or even trillions of parameters, the computational cost of full fine-tuning has become prohibitive (a single fine-tuning run can require thousands of GPU-hours). This has driven the industry toward a "frozen model + prompt optimization" approach, with representative work including Automatic Prompt Engineer (APE), EvoPrompt, and various reflection-based frameworks like Reflexion. These methods share a common trait: treating the model as a black box and modifying only the input text instructions to change output behavior, achieving performance gains at minimal cost.

However, existing reflective prompt evolution methods typically optimize the entire harness as a single flat string. Reflective prompt evolution is an automatic optimization paradigm that has emerged over the past two years. Its core mechanism has the LLM execute a task, generate natural language reflections based on the results, and feed those reflections back into the prompt — forming an iterative optimization loop. This essentially mimics the human "trial-error-reflect-improve" cognitive process, but traditional implementations treat the entire prompt as an indivisible optimization unit, lacking fine-grained management of different functional modules. The arXiv paper Where Does Harness-Optimization Value Live? raises a more fundamental question in this context: Where exactly does optimization value reside? The research team answers this through the HARNESSEVO method with a revealing finding — value is highly localized, and evenly distributing the optimization budget is actively harmful.

HARNESSEVO: Decomposing the Harness into Four Evolvable Slots

Unlike the traditional approach of optimizing the harness as a whole, HARNESSEVO decomposes it into four independently evolvable "slots":

• Role: Defines the agent's identity and persona
• Task-strategy: The overall methodology for solving problems
• Tool/format-rules: Output format and tool-calling constraints
• Reflection/control: Error reflection and control heuristics

Under an iso-budget setting, the researchers use the same reflective optimizer with "leave-one-in" and "leave-one-out" attribution analyses to precisely measure each slot's independent contribution. The leave-one-out approach removes one slot at a time to observe performance drops; the leave-one-in approach retains only one slot at a time to see how much performance it alone recovers. These are classic feature importance evaluation techniques widely used in feature selection and model interpretability. Used together, they provide a comprehensive view of each component's independent contribution and interaction effects — transforming the vague concept of "optimization value" into a quantifiable experimental metric.

Why Does the Overall Metric Show "No Improvement"?

Experiments were conducted in the ALFWorld environment using a frozen 7B model as the backbone. ALFWorld is a benchmark environment that combines text adventure games with embodied AI tasks, proposed in 2021 by Carnegie Mellon University and collaborators. Based on the ALFRED dataset, it converts household tasks that originally require visual input (such as "place the mug next to the coffee machine" or "heat the potato in the microwave") into purely text-based interactions. Agents must execute multi-step operations in a virtual home environment, including navigation, picking up objects, and using appliances. ALFWorld's core challenge lies in tasks requiring long-horizon planning, a large action space, and the fact that erroneous actions lead to ineffective exploration loops.

The results are initially surprising: HARNESSEVO's overall binary success rate is 0.657, while the original harness and flat-string evolution both score 0.642 — a negligible difference.

In other words, simply structuring the harness into slots does not automatically yield performance gains. The real insight lies in slot-level fine-grained analysis.

The inspiration for this structured slot design partially comes from the software engineering principle of Separation of Concerns — dividing a system into independent modules based on distinct responsibilities to facilitate maintenance and optimization. In traditional LLM prompt engineering practice, the system prompt is typically a long block of text mixing role, rules, strategy, and examples, with blurry boundaries between functions. HARNESSEVO's four-slot decomposition is essentially an engineering reconstruction of this chaotic state: the role slot corresponds to the motivational layer ("who am I"), the task-strategy slot to the planning layer ("how to do it"), the format-rules slot to the interface layer ("what format to output"), and the reflection/control slot to the error-correction layer ("what to do when things go wrong"). This layered architecture allows researchers to apply optimization pressure independently to each layer, enabling precise localization of how optimization value is distributed.

Key Finding: Optimization Value Is Highly Concentrated in the Reflection/Control Slot

Slot-level analysis reveals a clear conclusion: nearly all useful optimization value is concentrated in the reflection/control slot. This slot's "leave-one-in" gain is +0.119, while the other three slots (role, task-strategy, format-rules) contribute almost nothing independently.

This finding has significant implications for LLM agent optimization practice. It shows that for agent self-evolution, not all components are worth investing optimization resources in. Role definitions and format rules — components that might seem important — contribute almost no marginal improvement to final success rates on ALFWorld tasks. What truly determines success is whether the agent can effectively reflect on errors and adjust its control strategy. This conclusion is closely tied to the credit assignment problem — one of the core challenges in AI and reinforcement learning, first articulated by Marvin Minsky in 1961. In reinforcement learning, it refers to how to correctly attribute final reward signals to the specific actions or states that caused them. In the context of LLM agent prompt optimization, the credit assignment problem manifests as: when overall task success rates improve, should that improvement be attributed to better role definitions, strategy adjustments, or reflection mechanism improvements? HARNESSEVO successfully brings this classic problem into the prompt engineering domain through structured slot decomposition and attribution experiments.

Why Is the Reflection/Control Slot So Critical?

The logic behind this is worth exploring: ALFWorld is a multi-step interactive embodied task environment where agents frequently encounter reproducible, linguistically describable control failures (for example, repeatedly attempting ineffective actions, like repeatedly searching for an object in the wrong location). The reflection/control slot is precisely positioned to capture and correct these error patterns — encoding failure experiences as heuristic rules through natural language, enabling agents to proactively avoid repeating the same mistakes in similar situations. This mechanism is fundamentally similar to experience replay in reinforcement learning, except that experiences are stored and retrieved in text form. It's therefore natural that optimization gains concentrate here.

From a cognitive science perspective, the reflection/control slot functions like metacognition — "thinking about thinking." Psychological research shows that the core difference between experts and novices often lies not in the breadth of domain knowledge but in the strength of metacognitive monitoring: experts are more accurately able to identify when they've made mistakes, why they made them, and how to avoid repeating them. In the LLM agent context, the role and strategy slots provide "domain knowledge," while the reflection/control slot provides this metacognitive capability. ALFWorld tasks typically involve 10–50 consecutive actions; a single-step error can waste all subsequent steps. By encoding common failure patterns as explicit rules (such as "if searching the same location three times yields nothing, switch to another room"), the reflection/control slot effectively equips the agent with the judgment to know "when to stop and try a different approach."

The Budget-Splitting Trap: Why Uniform Allocation Is Actively Harmful

The paper's most cautionary finding is what it calls the "budget-splitting trap."

The researchers point out that distributing 64 rollouts evenly across four slots leaves each slot with only 16 — falling below the optimizer's "effective search threshold." This phenomenon can be understood from an optimization theory perspective: any sampling-based search optimizer (including evolutionary algorithms, Monte Carlo methods, etc.) has a minimum effective sample size — the minimum number of samples needed to find a candidate solution better than the current one with reasonable probability. This threshold depends on the search space size, the noise level of the objective function, and the optimizer's own efficiency. When the total budget is evenly split, if each subtask's share falls below this threshold, the optimizer gets stuck in a "sampling deficit" trap: the quality of generated candidate solutions becomes unstable and cannot reliably surpass the initial seed. Mathematically, this resembles insufficient exploration in the Multi-Armed Bandit problem — too few pulls per arm to accurately estimate expected returns.

The result is catastrophic: every slot freezes at its empty seed state, unable to produce any effective evolution. This explains why structured decomposition fails under uniform budget allocation.

Conversely, when concentrating the budget on the high-credit reflection/control slot — even using only half the split budget — the success rate recovers to 0.761. This contrast is compelling: optimization budget is not better when spread thin. Concentrating resources on the truly valuable components is what breaks through the optimizer's search threshold.

Implications for Self-Evolving Agent Architecture Design

This conclusion directly challenges the intuition that "more structure equals better performance." Decomposing a system into finer pieces might seem to give the optimizer more degrees of freedom, but it can actually dilute the limited search budget to the point where no part can be effectively explored. Credit assignment must precede structured evolution — identify where the value is before deciding how to allocate the budget. This principle aligns with the engineering management concept of the Pareto Principle (80/20 rule): most value typically concentrates in a small number of critical factors, and identifying and focusing on these factors is more effective than investing uniformly.

It's worth noting that "rollout" here refers to the process of an agent executing a complete task episode in the environment — from receiving the task instruction, through multi-step decision-making and action execution, until the task is completed or fails. Each rollout consumes inference computation resources (API calls or GPU inference time), so the number of rollouts directly constitutes the upper bound on the optimization compute budget. In reflective evolution frameworks, each rollout's result (success or failure with details) is fed back to the optimizer to generate improved prompt variants. In practice, 64 rollouts is already a fairly tight budget — at GPT-4-level API pricing, the multi-step interactions for each ALFWorld task can generate dozens of API requests, putting the total cost of 64 rollouts in the hundreds of dollars.

Optimization Effectiveness Is Strongly Correlated with Task Type

The researchers further validated on the WebShop environment and found a starkly different phenomenon: all slots froze to empty, and all methods tied. WebShop is a web shopping simulation environment proposed by Princeton University in 2022, where agents must search, browse, and purchase products on a simulated e-commerce site based on natural language shopping requirements (such as "find a black cotton crew-neck T-shirt in size L for under $25").

Notably, this freezing is not due to insufficient budget — it's because WebShop tasks lack reproducible, linguistically describable control failure patterns. Unlike ALFWorld, failure patterns in WebShop are typically one-time judgment errors (such as selecting the wrong product attribute) rather than repeatable control loop errors. The cause of failure in each shopping task tends to be different, lacking the systematic error patterns of ALFWorld's "repeatedly trying to open a locked door" variety. Without generalizable error patterns, the reflection/control slot has nothing to work with — the reflection mechanism loses the failure experiences it could encode and reuse.

This contrast emphasizes a core point: harness optimization value is not only localized but also task-contingent. In some task scenarios, the control slot is the key to improving performance; in others, it may have no effect at all. This also means that before deploying LLM agent systems in practice, conducting task characteristic analysis — particularly assessing whether the task contains generalizable systematic failure patterns — is a critical prerequisite for determining optimization strategy.

Summary: Optimization Strategy Insights for Self-Evolving LLM Agents

This paper provides three important conclusions for research and engineering practice on self-evolving LLM agents:

1. Harness optimization value is localized — optimization gains tend to concentrate in a small number of key components (the reflection/control slot in this study) rather than being evenly distributed across all modules. This aligns with the sparsity principle in machine learning: in high-dimensional spaces, the dimensions that truly affect the objective are typically few.
2. Uniform budget splitting can be actively harmful — spreading limited budget thinly across all slots causes each part to fall below the effective search threshold, resulting in across-the-board failure. This reminds us that under resource constraints, the design of the optimization strategy itself is a meta-problem that needs to be optimized.
3. Credit assignment should precede structured evolution — before decomposing and optimizing, identify where the value lies to efficiently allocate compute resources. The practical recommendation: use a small budget for attribution probing (e.g., leave-one-in/leave-one-out analysis) first, identify high-return slots, then concentrate the remaining budget there.

For developers building Agent frameworks, this means that rather than blindly applying large-scale automatic optimization across the entire prompt system, it's better to first conduct a "value localization" pass — directing precious compute budget toward the components that will actually deliver returns. In an era of increasingly complex LLM agent systems, this "focused" optimization strategy may be closer to the engineering optimum than trying to improve everything at once.

From a broader perspective, the findings also echo similar lessons from recent AutoML and Neural Architecture Search (NAS) work — blindly expanding the search space doesn't necessarily produce better results; what matters is the quality of the search space and the precision of budget allocation. This thinking extends to today's hot area of multi-agent system design: when multiple LLM agents collaborate to complete tasks, prompt optimization for different agents faces the same budget allocation problem, and prioritizing optimization of the bottleneck agent may be more efficient than uniformly optimizing all agents. An important direction for future research is developing adaptive credit assignment mechanisms — ones that can automatically identify which slots are most worth optimizing for the current task and dynamically adjust budget allocation strategy, rather than relying on manual upfront analysis.