Using AI Agents for Model Innovation: A Hands-On Guide from Ablation Studies to Performance Gains

Deep learning model improvement is often the most time-consuming and experience-intensive part of research and engineering — which modules to add, where to place them, how to combine them: every step demands extensive trial and error. Bilibili creator Qian Yun shares a complete workflow for using an AI Agent (Codex) to assist with model optimization. The core philosophy is "one step at a time" — using well-crafted prompts to guide the Agent toward genuinely improving metrics, rather than blindly stitching together modules. This article breaks down the key steps and practical details of this methodology.

Overall Approach to Agent-Assisted Model Optimization

This methodology uses a semantic segmentation task as its example, with the classic UNet as the base model and a starting mIoU of 0.817. The goal is to incrementally improve this metric with Agent assistance while keeping the process reproducible.

The author uses Codex as the demo tool, noting that Claude Code or other Agent tools work equally well. After installation, simply type codex in the terminal to open the interactive interface and drive the entire optimization process through conversation.

The critical first step is giving the Agent a clear anchor: tell it the current baseline metric (0.817), the model architecture (UNet), and the optimization goal — make the results better. This initial metric serves as the reference for all subsequent improvements. Without it, the Agent has no way to judge whether a change is actually effective.

Round 1: Training Strategy Optimization First

In the first optimization pass, the Agent's suggestions focused mainly on directional convolutions, deep supervision, and various training strategies. Test results showed: starting from 0.817, adding these changes pushed the score to 0.819, and combining deep supervision with directional convolutions brought it to 0.82.

Worth noting: when reviewing the logs, the author highlighted an easily overlooked fact — the gains in this round came primarily from training strategy adjustments, not core structural innovations. In other words, Agents excel at "hyperparameter-style" optimization, but producing research-worthy innovations still requires human guidance.

This explains why the author wasn't satisfied with the first-round results — limited metric gains and weak novelty. Model improvement must proceed step by step: first understand where each change's contribution comes from, then decide which direction to pursue next.

Deep Supervision is a training technique that introduces additional supervision signals at intermediate layers of the network, widely adopted in models like GoogLeNet. The core idea is to compute losses at different-scale branches of the encoder or decoder, rather than relying solely on the final output's backpropagated loss. This alleviates vanishing gradients and helps shallow-layer features train more effectively. In semantic segmentation, deep supervision is typically implemented as a weighted sum of the main loss and auxiliary losses, with auxiliary loss weights often decaying gradually during later training stages. Directional Convolution is a convolution variant designed for elongated structures (such as roads or blood vessels), extending the receptive field in specific directions to better capture anisotropic targets. Both of these modifications are lightweight adjustments to training and structure with low engineering overhead — which is why Agents tend to recommend them first. However, their ceiling for metric improvement is also relatively limited.

Round 2: Structural Innovation with a Module Library

To increase novelty, the author pre-loaded a Markdown document containing a large collection of plug-and-play modules (covering channel attention, spatial attention, multi-scale attention, and more) into the project folder, then prompted the Agent to select modules from it for experimentation.

After reviewing the document, the Agent selected three modules to enhance decoder feature representation. But the initial results were disappointing: starting from 0.816, adding EMA and LSK only reached 0.817 — a classic case of "ineffective stitching" where the modules were added but metrics barely moved.

This led to a key insight from the author: modules can't be added arbitrarily — placement is critical. He then provided more precise instructions: add EMA at encoder layers 2–3 and 3–4, and add a multi-scale feature aggregation module (ASPP) at the deepest layer. With clear placement guidance, EMA and ASPP both showed improvements individually and in combination.

If you're unclear about the model architecture or don't know where to place a module, you can first ask the Agent to analyze the model structure and identify optimal insertion points before proceeding. This step is the dividing line between "making the Agent effective" and "letting the Agent guess blindly."

EMA (Efficient Multi-scale Attention) is a lightweight attention mechanism that enhances joint channel-spatial representation through cross-dimensional multi-scale feature interactions, with significantly lower computational overhead than standard self-attention. ASPP (Atrous Spatial Pyramid Pooling), introduced by the DeepLab series, uses multiple parallel dilated convolutions with different dilation rates to extract multi-scale contextual information and fuse the results, expanding the receptive field without reducing resolution. ASPP is typically placed at the encoder's end (at the deepest feature map), where semantic information is richest and spatial resolution is lowest — making multi-scale receptive fields most beneficial. Arbitrarily inserting such modules in shallow layers can actually interfere with low-level texture feature learning, which was the main reason the initial attempt in this round underperformed. Understanding a module's design intent and appropriate layer placement is the theoretical basis for deciding "where to insert it."

Round 3: Combining Strategies and Modules

In the final round, the author integrated the results from both previous rounds: stacking the best training strategies from round one, the new network structure, and the innovative modules identified in round two.

Four directions were involved: directional convolution, deep supervision, EMA, and ASPP — theoretically yielding 16 combinations. The author tested them systematically with fixed hyperparameters. He stopped at combination 13, as the remaining 5 combinations offered diminishing marginal value — a pragmatic approach to saving experimental compute.

The final ablation study results showed that deep supervision, directional convolution, EMA, and the added modules each contributed individual gains — but it's important to note that certain combinations actually decreased performance (more isn't always better). Compared to the original weights, the final solution achieved an average mIoU improvement of approximately three points. The deep supervision approach was inspired by Ultralytics' related design, and the EMA attention mechanism contributed a notable gain compared to the original 0.817 baseline.

Ablation Study is the standard method in deep learning papers for verifying each component's contribution: change only one variable at a time while keeping everything else constant, attributing metric changes to that specific variable. The "group-by-group testing" described here is essentially a systematic ablation process. For N independent modules, a complete ablation requires 2^N combinations, growing exponentially with N. In practice, a greedy strategy is commonly used: first validate each module individually, retain effective ones, then perform pairwise or triple combinations — compressing the search space from exponential to near-linear. The author's decision to stop at combination 13 reflects this pragmatic tradeoff: once the main individual and higher-order interaction effects of key modules have been covered, the marginal informational value of remaining combinations is genuinely limited, and the compute cost of continued testing is hard to justify against potentially tiny gains.

Reusable Methodology Takeaways

Beyond the specific task, this workflow distills several generalizable principles:

• Baseline first: Give the Agent a clear starting metric and model information so every change has a reference point.
• Single-point validation before combining: Test each module's contribution in isolation before combining, to avoid confounding the source of improvements.
• Placement matters more than quantity: Where you insert attention and other modules directly determines success or failure. If uncertain, have the Agent analyze the architecture first.
• Use prompt templates strategically: Well-crafted prompts make the Agent's changes more precise while reducing token consumption.

The author mentions that accompanying prompt templates, the plug-and-play module document, and installation guides will all be provided. For researchers and engineers who need to "boost scores," this collaborative mode of "human sets direction, Agent executes" is far more reliable than letting the Agent freestyle on its own.

A note of caution: the metric gains shown in the video (approximately three points) are based on a specific dataset and UNet task. Actual results will vary by task — the methodology's value outweighs any specific numbers.