A Survey on Agent Self-Improvement: How Modern Agents Achieve Self-Evolution

As application scenarios grow increasingly complex, enabling agents to continuously improve their capabilities, self-correct, and absorb experience with minimal human intervention has become a critical challenge for deploying AI agents in the real world. Professor Gu Dandan from the School of Artificial Intelligence at Jilin University delivered a talk at NSAE's knowledge-sharing session, presenting her survey Self-Improvement in Modern Agentic Systems, co-authored with the AI Pioneer team at King Abdullah University of Science and Technology. The talk offered a systematic overview of this field's historical trajectory, formal definitions, and technical approaches.

What Is Self-Evolution: From Least Squares to Self-Induced Improvement

To understand self-evolution, it helps to start with the classic method of least squares. As a parameter estimation technique, least squares learns model parameters by minimizing an error function. Although a more complex model can fit complex data, its underlying function structure and optimization algorithm are predefined by humans. It can only passively accept external data and human-designed loss functions — it cannot audit or rewrite itself when its learning mechanism no longer fits the task.

This is precisely the boundary that self-evolution aims to break. The survey defines self-improvement as an update process induced by the system itself: as the system executes its current policy and interacts with tasks, it generates learning signals that are then used to make substantive modifications to the system's internal configuration, thereby altering the policy that governs its future behavior. The key phrase here is "self-induced."

This concept is far from new. Professor Gu traced its century-long evolution: in 1931, Gödel proved that formal mathematical systems can encode propositions about themselves and their own provability; later researchers proposed the idea of an "ultra-intelligent machine," arguing that once a machine could design a machine smarter than itself, an intelligence explosion would follow; von Neumann, drawing on automata theory, studied how a system could describe its own structure and achieve self-replication; and the Gödel Machine, proposed in 2003, established a theoretical framework in which a system may rewrite its own code — including the improvement mechanism itself — as long as it can formally prove that a given modification will increase its future expected utility.

The Gödel Machine is a foundational concept for understanding the theoretical basis of self-evolution. Proposed by Jürgen Schmidhuber in 2003, it is a fully self-referential, general problem-solving framework: the system treats its own complete code — including its search algorithms and learning mechanisms — as modifiable objects. As long as the system can construct a formal proof that a self-modification will improve its future expected utility, it may execute that modification, including modifying the mechanism used to construct the proof itself. The philosophical significance of this framework lies in bridging the "meta level" and the "object level" — self-improvement is no longer limited to weight adjustment but can extend to the system rewriting its own reasoning logic. Its limitation is that constructing formal proofs is computationally prohibitive, keeping the framework theoretical for decades. Large language models, with their natural language reasoning capabilities, now offer new possibilities for approximately realizing this framework.

Why It Matters Now: Large Models Breathe New Life into Self-Evolution

If the concept has existed for so long, why has it only gained real traction in the era of large language models? The answer lies in two fundamental bottlenecks of the pre-LLM era. First, models lacked broad world knowledge and strong reasoning capabilities, making it difficult for them to generate effective self-improvement strategies. Second, the modifiable substrate was severely constrained — traditional self-improvement approaches required searching through enormous low-level spaces such as raw neural network weights or assembly code representations, which was prohibitively expensive and ill-suited to open-ended tasks.

Large models changed this landscape. Trained on vast pretraining data and refined through repeated architectural and scaling improvements, LLMs possess powerful general-purpose knowledge and reasoning abilities, making them capable of "generating improvement strategies." More importantly, LLMs use natural language as a semantic interface between humans and models — natural language makes it straightforward to explicitly express and explain modifications: what changes were made are immediately transparent through language. When natural language becomes a shared semantic interface, self-evolution becomes far more tractable.

A Formal Decomposition of Modern Agent Systems

The survey defines a modern agent system as consisting of two parts: the parameters θ of a Foundation Model, and a Scaffold (also commonly referred to in the industry as a Harness). The scaffold is further broken down into prompts, memory, tools, and the overall control logic. When a user submits a request to the agent, the scaffold handles perception and interpretation, the foundation model acts as the "brain" for reasoning, the system interacts with the environment to obtain feedback, and results are produced.

Self-evolution is then formalized as an iterative process from time step t to t+1: the system uses the self-induced signal S and an improvement function to produce an evolved agent. The survey is organized along two main threads accordingly — evolution of the foundation model (Chapter 5) and evolution of the scaffold (Chapter 6) — with each chapter further categorized by "where the learning signal comes from."

Foundation Model Evolution: Internalizing Experience into Parameters

Evolution through modification of foundation model parameters is divided into three categories based on the source of the learning signal.

The first category is internally generated demonstration data. The model generates training data for the current task on its own, then internalizes that temporary experience through parameter updates. Representative work has models generate candidate "self-edits" in each iteration to update weights, then improve capabilities based on rewards. The risk with this approach is error accumulation — if the self-generated data is inaccurate, repeated iterations can lead to model collapse; the survey also discusses mitigation strategies.

The second category is self-generated feedback. The agent acts as a critic to produce evaluation signals, where the criteria may be explicit guidelines, consistency voting across multiple samples, or direct critique with suggestions for revision. The representative Test-Time Reinforcement Learning approach works by generating answers multiple times and taking a majority vote, using the vote result as a pseudo-label to compute rewards and update the model — enabling model updates even in the absence of ground-truth labels.

The third category is learning through action, where the agent interacts with the external world and uses the resulting feedback as experience. Interaction with real environments (such as code execution, the web, or GUIs) is costly, so an alternative path is learning from imagined proxy environments — for example, training a world model to substitute for the real environment. A 2026 robotics paper used a world model to generate learnable simulated experience, enabling the robot to self-improve on that basis and substantially reducing dependence on real-world interaction.

The strength of foundation model evolution is that it persistently internalizes experience into weights, enabling broad and lasting capability improvement. The cost is high computational expense and a strong dependence on the quality of the self-induced signal.

Scaffold Evolution: Flexible Adaptation at Low Cost

Scaffold evolution does not modify model parameters; instead, it modifies prompts, memory, tools, and even the overall workflow. The survey organizes this into four sections.

Prompt evolution is divided into four types based on feedback signal: scalar feedback (optimizing prompts based on accuracy or preference scores), qualitative feedback (optimization using natural language content such as critiques and error analyses), population-based crossover and mutation for exploration, and text gradient methods. TextGrad, published in Nature in 2025, treats natural language feedback directly as "text gradients" flowing backward through the system — without computing a real loss, optimization signals can propagate through the language system to guide iterative refinement of prompts and intermediate components.

Memory evolution is explored from three angles: what to remember, how to organize it, and how to process it. Organizational forms have evolved from flat storage to hierarchical structures (distinguishing between general knowledge and short-term memory) and even graph structures. Processing mechanisms involve create, read, update, and delete operations — analogous to how human memory requires selective forgetting. Representative work transforms memory from static storage into a dynamic system that continuously organizes, updates, and links content as new experiences arrive.

Tool evolution encompasses tool routing (autonomously selecting needed tools), tool refinement (modifying tools that don't work well), and tool creation (building new tools when existing ones are insufficient). Work from 2025 can standardize discovered code and capabilities into reusable tools, allowing one-off solutions to be deposited into a tool library for future reuse.

Full scaffold evolution treats everything beyond model parameters as a modifiable program, restructuring workflows and control logic. This is the most difficult but also the most flexible approach. The most representative task is coding agents. Professor Gu's team built on the Gödel Machine framework for their ICLR paper, treating the entire scaffold of a coding agent as an evolvable program and designing an evolution strategy by evaluating the long-term potential of each modification node.

The advantage of scaffold evolution lies in its speed, flexibility, and explicit modifiability — modifications are easy to inspect, users can intervene at any time, and changes can be rolled back to a prior version. Full scaffold modification, meanwhile, moves toward deeper, introspective structural reconstruction.

TextGrad is an important reference for understanding the new paradigm of "text gradients." In traditional deep learning, gradients are partial derivatives of a loss function with respect to parameters, propagating numerically through the chain rule layer by layer to guide parameter updates. TextGrad transfers this idea to language systems: when a component (such as a prompt, a reasoning step, or a code snippet) produces a suboptimal output, an LLM generates natural language criticism as a "text gradient" and propagates it backward to upstream components to guide their revision. This way, even when the entire pipeline is non-differentiable and lacks ground-truth labels, optimization signals can flow between components in linguistic form. Its core value lies in extending the applicability of "backpropagation" from the continuous parameter space to the discrete language space, enabling systematic iterative improvement of components — such as prompts, memory contents, and tool descriptions — that were previously inaccessible to gradient-based optimization.

The Core Bottleneck: How to Verify That an Improvement Is Actually Effective

During the Q&A session, Professor Gu identified the biggest bottleneck in self-evolution today as verifying whether an improvement is actually effective. In closed, executable, and quantifiable tasks like coding, you can simply run test cases after making changes to judge quality. But in open-ended tasks, it is often very difficult to determine in the moment whether the current improvement is better than the previous one — evaluation is both complex and expensive. Even for closed coding tasks, evaluating an improvement across an entire benchmark is far more costly than proposing the improvement itself, and when real-environment interaction is involved (such as robotics), the cost of verification rises steeply.

Another issue raised in the discussion is the temporal validity of improvements: a modification verified as beneficial at time t may actually cause negative effects at t+1 or t+2. This requires the system to have a rollback capability — the ability to assess whether historical modifications remain beneficial for the future and to revert when necessary. Professor Gu candidly noted that when the team first began writing the survey, they found almost no work in this area, but it is genuinely an important direction that deserves more attention.

The verification problem has deep historical roots in reinforcement learning, corresponding to the challenges of reward sparsity and evaluator bias. In closed tasks, execution results are immediately observable and reward signals are dense and clear. In open-ended tasks, rewards are often sparse or absent, and the system must rely on the model itself or another model to act as a judge or critic. However, when the improver and the evaluator come from the same model family, a "self-serving bias" emerges — models tend to rate their own outputs highly. In the context of self-evolution, this is compounded by distribution shift: as the agent's capabilities evolve, its behavioral distribution changes, rendering earlier evaluation benchmarks potentially obsolete, meaning the system ends up "measuring its old self" rather than its current capabilities. This is precisely why rollback mechanisms and the design of continuously valid evaluation benchmarks have become critical research topics.

Co-Evolution and Future Directions

On the question of whether scaffold and parameter evolution should proceed in separate layers, Professor Gu noted that the survey separates them primarily for clarity of exposition. In practice, the two can alternate: scaffold evolution requires fewer resources and is faster, making it suitable for initial exploration; when accumulated prompts and memory banks become excessively redundant and experience is "about to explode," these external constraints can be distilled and internalized into the model's parameters. This is consistent with the broader trajectory of LLM development — the more capable the model, the less prompting and context it requires.

The survey's guidance on practical applicability is: self-improvement is most suitable when the agent can obtain executable, quantifiable feedback. Looking ahead, promising directions include test-time continual adaptation, curiosity-driven active exploration, co-evolution of parameters and scaffolds along with attribution methods, cost-benefit tradeoffs under resource constraints, multi-agent systems, and handling distribution shift in open-world settings. Safety drift and loss of control, the tradeoff between exploration cost and benefit, and reliable internal evaluation judges are identified as the most critical research challenges.

Professor Gu's team maintains a continuously updated GitHub repository with relevant work organized by chapter, and welcomes community contributions.