How Self-Improving AI Agents Learn from Experience

As frontier labs race toward more powerful foundation models, a more grounded question is emerging: how can AI agents, like human employees, learn from every execution and get smarter the more they're used?

From Static Prompts to Experience-Driven

Traditional agents work in a simple way: receive a prompt, call tools, return results. Task over, everything resets—the same mistake may be repeated next time, and the same edge cases must be figured out all over again. This pattern looks impressive in demos, but in production, every failure means valuable debugging effort is wasted.

The core idea of self-improving agents is to break this "one-shot" cycle. It introduces a key assumption: the agent's execution trajectory itself is training data. When a task fails, the reason for the failure isn't the endpoint—it's the starting point for the next round of practice. Two elements are at work here: executable tasks define what counts as "doing it right," while an adaptive curriculum decides what to practice next.

Execution trajectory is a foundational concept for understanding self-improving agents. It refers to the complete sequence of records produced by an agent while completing a task, typically including: the observation at each step (the state returned by the environment or tool output), the thought (reasoning chain), the action (tools invoked and parameters), and the final result. In traditional reinforcement learning, agents rely on immediate reward signals from the environment to learn; the unique value of the execution trajectory is that it preserves the full context surrounding a failure—which reasoning step went wrong, which tool call used the wrong parameters, which prior assumption didn't hold. This fine-grained information carries more learnable signal than a simple "failed" label. Thus, treating execution trajectories as training data is essentially extending from "outcome supervision" to "process supervision," enabling agents to extract lessons from the specific step where the error occurred, rather than merely knowing the task ultimately wasn't completed.

Adaptive curriculum learning originates from the idea of curriculum learning in educational psychology and machine learning. Classic curriculum learning was proposed by Bengio et al. in 2009, with the core claim that training samples should be presented to the model in order from easy to hard, rather than randomly shuffled. Adaptive curricula further introduce a feedback loop: difficulty isn't fixed in advance but dynamically adjusted based on the model's current performance. In educational psychology, this echoes Vygotsky's "Zone of Proximal Development" theory—learning content should be slightly above the learner's current ability to complete independently, but achievable with guidance. For agents, this means the success rate can't be too high (no room to learn) or too low (feedback signals drowned out by noise); the ideal success rate range is typically set between 50% and 80%, with specific thresholds varying by task type.

Executable Tasks: Turning Vague Goals into Verifiable Signals

One of the biggest learning obstacles for agents is that reward signals are too vague. When a human says "help me analyze this market," what counts as a good analysis? What counts as failure? Without clear criteria, an agent has no way to know whether it's improving.

Executable tasks eliminate this ambiguity. An executable task typically consists of three parts: a preset initial environment, verifiable success conditions, and an allowed action space. For example, "fix the function that's causing this unit test to fail" is an executable task—whether the test passes is a hard metric with no room for dispute.

This design gives the agent's trial-and-error a direction. Each execution either satisfies the conditions or doesn't, with clear feedback. More importantly, failure information can be recorded in a structured way: which step caused the state to deviate, what prerequisite knowledge was missing, whether the tool-call order was reasonable. If effectively used, this information can be turned into material for subsequent training.

In the fields of reinforcement learning and agent evaluation, verifiable success conditions correspond to the role of a "reward function" or "verifier." A well-designed executable task must have success conditions that are deterministic and automatically judgeable—for example, whether code tests pass, whether a math answer matches, whether a database state reaches expectations. This is fundamentally different from approaches that rely on human subjective scoring or another model's grading: the latter introduces extra noise and uncertainty and is prone to reward hacking in repeated iterative training loops—the agent learns to satisfy the grader rather than actually solve the problem. Therefore, selecting executable tasks often prioritizes scenarios with objective verification mechanisms, such as unit tests, formal proofs, and API return-code checks. This also explains why current success stories of self-improving agents are mostly concentrated in programming, mathematical reasoning, and structured operation tasks, while open-ended creative or strategic decision-making tasks remain hard to fit into this framework.

Adaptive Curriculum: Letting Practice Difficulty Follow Ability

Having executable tasks isn't enough. If all tasks are uniformly distributed in difficulty, an agent may repeatedly succeed on easy tasks without learning anything new, or repeatedly fail on hard tasks without receiving effective feedback. Adaptive curricula solve exactly this problem.

Its basic logic is: dynamically adjust the difficulty and type of the next training task based on the agent's current ability level. If the agent's success rate on a certain type of task stably exceeds a threshold, the curriculum introduces harder variants or new task types. If the success rate drops sharply, the curriculum falls back to tasks of similar difficulty, letting the agent consolidate the basics first.

The intuition behind this is much like human learning. You wouldn't have a student who just learned addition and subtraction do calculus, nor would you have someone who has already mastered calculus repeatedly do addition and subtraction. Adaptive curricula keep the agent's practice in the zone of "reachable with a stretch," maintaining both learning efficiency and motivation.

A Closed Loop from Production Failures to Useful Practice

Putting executable tasks and adaptive curricula together forms an experience-driven learning loop:

1. Execute: The agent completes a task in a real or simulated environment
2. Judge: Determine the result based on the executable task's success conditions
3. Attribute: Analyze the cause of failure and extract reusable experience fragments
4. Update curriculum: Adjust the difficulty and type of subsequent practice based on performance
5. Retrain: Update the agent's policy using accumulated experience data

The key to this loop is that failures in production are no longer pure loss. Every failure is converted into practice material that the curriculum system can schedule. Edge cases, tool-call failures, and broken reasoning chains that the agent encounters in real scenarios all enter the subsequent learning process in some form.

This approach is clearly different from simply "increasing data volume." The latter assumes more data leads to better performance, while experience-driven methods focus more on the structure and timeliness of data—not all failures are equally important, and not all experiences are worth repeatedly practicing. Choosing what to practice, how much to practice, and when to practice is itself part of improving the agent's capabilities.

How Far Is This Path?

This approach still faces many challenges. Attribution analysis itself is difficult—when a task fails, is it the model's reasoning that went wrong, inaccurate observation of the environment state, or an unstable tool interface? Accurately distinguishing these factors requires additional diagnostic mechanisms.

Curriculum design also involves trade-offs. Overly aggressive difficulty increases may trap the agent in consecutive failures, while overly conservative ones waste training opportunities. How to design appropriate difficulty gradients and success thresholds for different types of tasks still has no universal answer.

But the direction is clear: let agents learn from every execution, rather than starting from zero. When executable tasks provide reliable learning signals and adaptive curricula ensure the effectiveness of practice, experience-driven agents have a foundation for continuous evolution in complex environments.

One-line Summary

Self-improving agents turn execution trajectories into training data, using executable tasks and adaptive curricula to build an experience-driven learning loop.

Paragraph Summary

This article explores how self-improving agents can learn from every execution like human employees. The core idea is to break the traditional agent's "one-shot" working mode and treat execution trajectories themselves as training data. Two key mechanisms support this loop: executable tasks provide clear, undisputed feedback signals through preset environments, verifiable success conditions, and action spaces; adaptive curricula dynamically adjust practice difficulty based on the agent's current ability, keeping training in the zone of "reachable with a stretch." Together they form a complete loop of execution, judgment, attribution, curriculum update, and retraining, turning failures in production from pure loss into schedulable practice material. The article also notes that current main challenges lie in the accuracy of failure attribution and the design of curriculum difficulty gradients.