AIBuildAI-2.5: Breaking the Efficiency Bottleneck in Autonomous Modeling with LLM-Guided Tree Search

Autonomous AI agents are transforming how machine learning models are built. If a system can automatically handle the full pipeline—from data processing to model training—AI capabilities could spread into far more areas of science and engineering. A new paper on arXiv introduces AIBuildAI-2.5, an autonomous modeling system powered by large language model (LLM) agents. It specifically targets three efficiency weaknesses in existing approaches and sets a new record on a leading benchmark.

Framing Modeling as a "Code Search" Problem

One popular class of autonomous AI agents abstracts model building as a code search problem and solves it via tree search. In this search tree, each node is a candidate program, and the tree grows by generating an improved child program from a parent node. Using this approach, such agents have already achieved performance on real-world benchmarks that rivals experienced AI engineers.

Yet beneath this seemingly mature paradigm lie structural efficiency problems. The paper explicitly identifies three weaknesses that have not been adequately addressed—and these three points largely determine whether an agent can operate effectively within a constrained compute budget.

Three Efficiency Bottlenecks

The "Noisy Scoring" Problem Under Limited Budgets

The first issue lies in evaluating candidate nodes. Within realistic compute budgets, the number of candidate programs that can actually be executed is very limited. Strategies like Monte Carlo-style tree search, which rely on executed rewards to rank nodes, must make decisions based on extremely few and noisy scores. Too few samples, too much noise—this makes the system inefficient when choosing the next direction to explore, and it easily goes down the wrong path.

Lack of Resource-Aware Job Scheduling

The second problem is in the scheduling of training jobs. Existing agents generally do not employ resource-aware scheduling strategies; training tasks are launched without regard to the actual state of available hardware. This directly reduces hardware utilization and overall training efficiency—a waste that cannot be ignored in real-world environments where GPU resources are scarce and costly.

Inflated Inference Costs from a Single Large Model

The third issue is cost. Current approaches use the same powerful model for every agent call. Whether the task is simple or complex, the top-tier model is always invoked, which drives up inference costs significantly. For a search process that may require tens of thousands of calls, this "sledgehammer for a nail" approach is economically unsound.

AIBuildAI-2.5's Three-Pronged Solution

AIBuildAI-2.5 offers a corresponding engineering solution to each of the three problems, forming a complete agent system.

LLM-Guided Tree Search

The core innovation of the system is a novel "LLM-guided tree search." Rather than relying purely on post-execution reward scores, it introduces a judge role that evaluates each candidate program along three dimensions: expected improvement, grounding, and feasibility. A selector then combines these scores with the current search state to rank the candidate pool. This approach allows the system to leverage the LLM's prior judgment to filter directions before programs are executed, alleviating the "noisy scores, few samples" dilemma.

The key value of incorporating an LLM into search decisions lies in its ability to make "execution-free" prior judgments. Traditional MCTS must run the code first to obtain a reward signal, whereas the LLM judge can perform multi-dimensional evaluation based on the code text alone, before execution. "Expected improvement" measures whether the proposed solution offers a substantive advance over the current best; "grounding" checks whether the modification has a sound machine learning rationale rather than being a random perturbation; "feasibility" assesses whether the code can run to completion within the given compute constraints. Together, these three dimensions form a soft filtering layer, enabling the system to prioritize the most valuable candidates and allocate limited GPU time toward the most promising directions. This "judge first, execute later" paradigm is essentially grafting the LLM's semantic understanding onto the node selection step of a traditional search framework—a key design choice that allows search to converge efficiently even under sparse reward conditions.

Resource-Aware Scheduler

To address hardware utilization, AIBuildAI-2.5 incorporates a scheduler that takes current hardware resource status into account when launching training jobs. This resource-aware scheduling aligns compute allocation more closely with actual workload, thereby improving training efficiency.

Cost-Aware Model Router

To combat inference cost inflation, the system features a router that assigns lower-cost LLMs to less demanding tasks while reserving the most powerful model for the most challenging subtasks in the modeling pipeline. This tiered dispatching approach maintains quality at critical steps while significantly reducing overall inference expenditure.

Benchmark Performance

Results ultimately speak for themselves. The paper reports that AIBuildAI-2.5 ranks first on MLE-Bench, achieving a medal rate of 73.3%. It also outperforms a strong baseline system on six autonomous AI research tasks from AIRS-Bench.

The medal rate metric reflects the proportion of competition-style machine learning tasks on which the agent achieves a medal-level result. A score of 73.3% means the system produces competitive model solutions on the majority of tasks.

MLE-Bench is a standardized evaluation framework proposed by OpenAI that adapts historical Kaggle competition problems—spanning image classification, time-series forecasting, natural language processing, and more—into an automatically scorable benchmark. It uses historical competitor results as reference lines and defines gold, silver, and bronze medal thresholds accordingly. The medal rate thus becomes a comprehensive metric that tests both the upper bound of model capability and cross-task generalization, making it much harder to game through overfitting than single-task accuracy. AIRS-Bench, on the other hand, focuses on more open-ended autonomous AI research scenarios, requiring the agent to autonomously complete a full loop of literature review, solution design, and experimental validation given a research objective—evaluating end-to-end research capability rather than isolated technical skills. The complementary nature of these two benchmarks makes the conclusions drawn from evaluating AIBuildAI-2.5 more convincing: the former validates engineering execution quality, the latter validates research autonomy.

Implications for Autonomous AI Development

The value of AIBuildAI-2.5 lies not in a single technical breakthrough, but in the fact that it systematically places "efficiency" at the center of autonomous modeling. Previously, this class of agents competed primarily on capability ceiling—can they produce a good model? This work shifts the focus to how to produce good models quickly and cheaply within real budget constraints.

Introducing LLM prior judgment into search decisions, making schedulers hardware-aware, and routing models by task difficulty—these three improvements all point in the same direction: making autonomous AI development more practical and deployable under limited resources. For the goal of lowering the barrier to AI modeling and broadening AI's application in science and engineering, this kind of efficiency optimization may be more practically meaningful than simply throwing more compute at the problem.

Background

Tree search, as a classic discrete optimization framework, is used in autonomous modeling to traverse "program space." Starting from a parent node each time, the LLM generates several variants as child nodes based on the current code; the system then selects the most promising branch to expand according to scores, until a sufficiently well-performing program is found. Monte Carlo Tree Search (MCTS) is the most common variant, with the core idea of estimating the long-term value of each node through large numbers of random simulations, then using strategies like Upper Confidence Bound (UCB) to balance exploiting known good nodes versus exploring unknown directions. This framework has been thoroughly validated in board games (such as AlphaGo), but when transplanted to code generation it faces a critical difference: simulation cost in games is extremely low, allowing rapid accumulation of many samples; executing a machine learning training script, however, often takes minutes to hours, and with few samples, noise significantly distorts ranking decisions.