AREX-2: Building Self-Improving AI Agents with Long-Horizon Reflective Tasks

Why Self-Improving Agents Are a Critical Breakthrough

The capability frontier of LLM agents is evolving from "answering on the first try" to "iteratively refining answers." A recent paper on arXiv introduces AREX-2, whose core goal is to advance self-improvement in LLM agents — that is, the ability to iteratively optimize solutions at test time.

This capability sounds straightforward, but achieving it relies on two complementary underlying mechanisms. The research team breaks them down as: reflection, which produces a solution better than the current version; and long-horizon execution, which keeps the iteration loop effective across many rounds rather than stalling or degrading after just a few. Neither can stand alone — without reflection, iteration has no direction; without long-horizon execution, even the best reflection cannot accumulate into genuine progress.

The Core Hypothesis: Capabilities Are Domain-Agnostic

Underpinning AREX-2 is a bold and crucial hypothesis: reflection and long-horizon execution are domain-agnostic. In other words, if an agent learns "how to reflect and how to keep iterating" in a setting well-suited for supervised training, that skillset can transfer to entirely different task domains.

The value of this hypothesis lies in solving the data sourcing problem. If the capabilities themselves are general, training data does not need to cover every target domain — it can instead be synthesized in the settings where verifiable feedback is easiest to obtain. This is exactly the approach the research team took, selecting two categories of tasks with clear feedback signals to generate training data.

Synthesizing Training Trajectories from ML and Algorithmic Programming

The team synthesizes long-horizon improvement trajectories from two domains: machine learning tasks and algorithmic programming tasks. Both share a key advantage — they provide verifiable feedback and naturally reward continuous iteration.

In machine learning tasks, model performance can be quantified through metrics, giving clear signal on whether each round of tuning is an improvement. In algorithmic programming tasks, test cases verify both correctness and efficiency. This verifiability allows agents to genuinely learn, during training, "what kinds of modifications lead to improvement" — internalizing reflection and long-horizon execution as transferable capabilities.

A long-horizon improvement trajectory is a complete sequence recording an agent's journey from an initial solution through multiple rounds of reflection and revision until converging on a better solution. Unlike single-step Q&A data, these trajectories capture intermediate reasoning steps such as "why the previous version fell short," "what needs to change," and "how the revision performed" — enabling the model to learn iterative improvement strategies rather than just final answers. The key challenge in synthesizing such data is having a reliable external feedback signal to determine whether each modification truly constitutes progress. Evaluation metrics in machine learning tasks (e.g., validation set accuracy) and test cases in algorithmic programming (e.g., correctness and runtime efficiency) provide exactly this kind of objective, automated feedback, making large-scale trajectory synthesis feasible while eliminating dependence on manual annotation.

The hypothesis of domain-agnostic transfer has deep theoretical roots in machine learning, aligning closely with the core ideas behind transfer learning and meta-learning. The intuition is that certain cognitive abilities — such as "identifying errors, proposing improvements, and verifying whether improvements hold" — are meta-capabilities shared across tasks rather than domain-specific knowledge. A human analogy: a programmer trained extensively on debugging code can often transfer that systematic problem-diagnosis mindset to writing revision or experimental design. AREX-2's novelty lies in making this hypothesis explicit and putting it to empirical test: choosing two domains with the clearest feedback signals as "capability incubators," then measuring transfer through cross-domain benchmarks. The experimental results provide strong empirical support for the hypothesis, though its boundaries — for instance, whether transfer remains equally effective in domains requiring highly specialized expertise — remain open questions for future research.

Experimental Results: From Training Domains to Broad Transfer

AREx-2 is built on Qwen3.8-27B and achieves strong results across multiple benchmarks. In training-related domains, it scores 81.8 on MLE-bench Lite and 70.7 on Frontier-CS, validating the effectiveness of synthetic trajectory training.

Even more telling is its cross-domain transfer performance. Despite being trained on machine learning and algorithmic programming data, AREX-2 transfers effectively to deep research tasks, with scores including:

• BrowseComp: 84.0
• HLE: 52.6
• GAIA: 92.2
• DeepSearchQA: 93.8

These numbers directly support the core hypothesis that capabilities are domain-agnostic — the reflection and long-horizon execution abilities acquired in programming and machine learning settings do indeed transfer to information retrieval, complex reasoning, and other fundamentally different tasks.

More Iteration Budget, Better Performance

The paper also highlights a notable property: the agent continues to improve as the budget of rounds increases. This means AREX-2 does not hit a ceiling after a few iterations — it can effectively leverage additional test-time compute to achieve better results.

This aligns closely with the field's growing interest in test-time scaling. If an agent can reliably grow stronger with more rounds of deliberation, it provides users with a controllable quality-cost tradeoff lever: invest more rounds in high-value tasks to obtain better solutions.

Test-time scaling is an important trend in recent LLM research. Unlike training-time scaling (scaling up parameters or training data), test-time scaling refers to investing more compute at inference — for example, increasing thinking steps, sampling more candidate answers, or executing more iteration rounds — to obtain higher-quality outputs. The success of OpenAI's o1/o3 series and models like DeepSeek^Ad-R1 has already demonstrated that "thinking one step further" at inference time can significantly improve accuracy on complex tasks. AREX-2 extends this idea to the agent setting: the number of iteration rounds itself becomes an adjustable compute budget, and the returns scale monotonically with rounds rather than saturating prematurely. This means the framework is naturally compatible with a "pay-as-you-go" deployment model — use fewer rounds for time-sensitive lightweight tasks to return results quickly, and invest additional rounds in high-value complex tasks to unlock better solutions, creating a controllable quality-cost tradeoff.

Implications and Outlook

The core conclusion from AREX-2 is clear and compelling: long-horizon reflective data is an effective path toward self-improving agents. It demonstrates that rather than collecting data directly from target domains, one can achieve broad capability transfer by training reflection and long-horizon execution in settings with abundant verifiable feedback.

For the broader agent research community, this offers a replicable methodology: first identify which capabilities are general, then efficiently cultivate them in the domains most amenable to supervision, and finally rely on transfer to spread those capabilities into settings that are difficult to train on directly. As more and more agents begin emphasizing continuous optimization at test time, work like AREX-2 may become an important reference point for future research.