AQuA Model-Search Agent: Why the Evaluator Must Stay Sealed

A Deliberately Constrained Model-Search Agent

When an AI agent can rewrite both the model itself and the entire experimental pipeline surrounding it, a higher score tells you nothing — you can't tell what actually drove the improvement. This is precisely the design tradeoff worth examining in the second part of the AQuA paper: the "model-development loop."

Unlike the common approach of letting an agent freely generate training code and then just looking at the final metrics, AQuA does not allow the agent to produce arbitrary training code each iteration. Instead, every iteration can only propose a bounded configuration change within four restricted areas:

• Architecture
• Loss function
• Sampler
• Optimizer

The generated model is then trained under a sealed evaluator. Data splits, feature definitions, label definitions, and evaluation logic are all outside the agent's reach.

Redefining the Experimental Unit

This constraint fundamentally changes what an "experimental unit" means. A proposal is no longer "here's a new Python project, trust the final metric" — it's closer to a traceable formula:

Last accepted config + declared config diff + fixed training/evaluation framework = next candidate

The core value of this design is auditability. It doesn't automatically guarantee that every comparison is perfectly fair — architectural changes can still alter compute requirements, and different loss functions or optimizers may need different tuning. But it makes "what was changed" inspectable. When a result shifts, you at least get a bounded diff to audit, rather than facing an opaque mixture of model logic, data pipelines, labels, and metrics all tangled together.

Why a Sealed Evaluator Is So Critical

A long-standing risk in AutoML and agent-driven research is that when an agent can modify both the model and the evaluation method, it can easily "cheat" — quietly relaxing evaluation criteria or adjusting data splits to inflate scores. AQuA's approach essentially draws a clean line between the proposal language and the evaluator, fundamentally cutting off this path to attribution contamination.

Machine learning competitions and automated research have plenty of precedent here. The Kaggle community observed "leaderboard overfitting" long ago: competitors repeatedly submit to reverse-engineer the test set's distribution, then suffer severe drops on the private leaderboard. In an agent setting, this problem is amplified — the agent's optimization objective is to maximize observed evaluation scores, and if the evaluation itself is a manipulable variable, the gradient points toward "making metrics look good" rather than "making the model better." In information-theoretic terms, this is equivalent to leaking test set information back into the training/search loop, breaking the unbiasedness assumption underlying generalization error estimation. By making the evaluator an immutable "external judge," AQuA enforces information isolation between the search process and the evaluation process at the engineering level.

Time Splits Are Locked Down Too

In the paper's US-equity experiments, temporal handling is equally strict:

• Models train on 2010–2019 data;
• Early stopping and checkpoint selection use only an internal validation slice within the training window;
• 2020 is an "embargo zone" — untouched by both training and model selection;
• 2021–2025 serves as the final test window.

It's worth emphasizing that finance is simply the experimental domain chosen for this research — this is not trading advice. The deep learning question with transferable value is: how much freedom should an architecture-search agent be given before comparisons across experiments lose a consistent meaning?

Look-ahead bias in financial time-series data is one of the most common causes of evaluation failure. If training data or model selection inadvertently references future information — for example, using full-sample means for normalization, or implicitly referencing the test period's distribution when selecting checkpoints — backtest returns can be inflated many times over. AQuA's 2020 embargo zone is a standard practice of inserting a temporal buffer between the training and test windows, ensuring that models cannot gain a predictive edge over 2021 and beyond by "memorizing" the extreme market conditions of 2020 (the COVID shock). This design is a simplified version of what quantitative researchers call "purged cross-validation" — the core idea being that adjacent time-period samples are autocorrelated and must be actively isolated to ensure validation set independence.

Honest About Limitations

To its credit, the preprint is candid about its own verification limits.

First, the paper does not disclose the exact feature set, normalization approach, or label construction details. This makes it difficult for external researchers to fully reproduce or deeply audit the experimental setup.

Second, a constrained configuration interface does not mean every candidate model receives the same wall-clock compute. Some architectures may be "heavier" and consume more compute — this remains a potential source of unfairness.

What it does provide is a clean boundary between the proposal language and the evaluator — and that kind of engineering discipline is genuinely rare in agent-driven research today.

The Tradeoff Between Attribution Clarity and Search Freedom

The question AQuA raises is both highly discussable and cuts straight to the core tension in AI research automation:

Would you keep the search language this narrow to preserve attribution clarity? Or would you let the agent modify schedulers, preprocessing, and even training code — as long as every change is fully tracked?

This is fundamentally a tradeoff between expressiveness and interpretability. A narrower search space sacrifices the agent's creativity in exchange for attributable improvements; a wider search space might uncover more surprising solutions, but at the cost of never being able to say clearly "what actually worked."

For serious scientific work, an improvement that cannot be attributed is often worth significantly less. AQuA chose discipline over cleverness — which may not be the "smartest" path, but is probably the most credible one.

Preprint: arxiv.org/abs/2608.12841

This tension has been deeply explored in the neural architecture search (NAS) field. Early NAS work — NASNet, DARTS, and others — had opaque search space designs, and the architectures they "discovered" were later shown to depend heavily on specific proxy tasks and search budgets; change the evaluation setup and the conclusions shift. Some researchers have since argued that "the search space itself is a prior": researchers inject substantial domain knowledge when designing the search space, and an agent's "creative discoveries" are often nothing more than enumeration within human-defined boundaries. AQuA makes this implicit assumption explicit — it doesn't claim the agent can leap beyond the four human-defined dimensions, but focuses on doing auditable, systematic exploration within them. That is actually a more honest scientific posture.