LLM as a Universal Searcher: Open-World AutoML Without Fine-Tuning

From Domain-Specific to Universal Search: A Paradigm Shift in AutoML

Traditional AutoML frameworks have long been constrained by domain specificity—model architecture, training hyperparameters, and data mixtures are often optimized in isolation, making joint tuning difficult. A recent study proposes using large language models (LLMs) as universal searchers to achieve joint optimization of all three for the first time, without task-specific fine-tuning. The method, called NCode, transforms diverse AutoML design choices into discrete sequences through a unified numerical encoding and reframes the optimization process as a unified ranking task.

The core value of this approach lies in using an LLM searcher trained on optimization trajectories from a single vision task to handle cross-task, cross-modality scenarios such as image classification, image segmentation, speaker verification, and text-to-image generation, and to transfer to joint search spaces. This breaks the convention that AutoML requires redesigning the search space and training pipeline for every new task.

AutoML (Automated Machine Learning) aims to automate processes like model design and hyperparameter tuning that traditionally require repeated manual trial and error. Conventional AutoML typically decomposes the problem into separately handled sub-problems: Neural Architecture Search (NAS) focuses on network structure, Hyperparameter Optimization (HPO) focuses on training parameters like learning rate and batch size, and data mixture gets its own separate pipeline. This fragmentation has two consequences: first, searches in each subspace are independent, making it hard to find optimal combinations across dimensions; second, every time the task changes (e.g., from image classification to a speech task), the search space and search algorithm often need to be redesigned, resulting in poor generality. Early methods in this field relied on evolutionary algorithms, Bayesian optimization, or reinforcement learning, with high search costs and limited transferability. In recent years, researchers have begun trying to use pretrained language models as "universal searchers," encoding different design choices into the same sequence to complete joint search within a single framework—this is precisely the paradigm shift context that NCode represents in this article.

Diffusion Transformer (DiT) is a generative model architecture that introduces the Transformer into diffusion models, commonly used for tasks like text-to-image: the diffusion process progressively denoises from noise to generate images, while the Transformer serves as the backbone of the denoising network, replacing the U-Net commonly used in earlier diffusion models. DiT's parameter count, layer count, attention configuration, and other factors significantly affect generation quality, making it an ideal target for AutoML joint search. The "billion-parameter scale" mentioned in the article means the searcher must make decisions in an enormous configuration space with high per-training cost, which explains why the 896 GPU-days search expense is an important reference for evaluating the method's practicality.

Cross-Task and Cross-Modality Generalization

The research team trained the LLM searcher using only optimization trajectories from limited, single-domain vision tasks. Results show that the searcher achieves state-of-the-art performance on image classification, image segmentation, speaker verification, and text-to-image benchmarks. This means the searcher learned not task-specific tricks but a transferable optimization strategy.

Cross-modality generalization is particularly noteworthy. Speaker verification belongs to the audio domain, and text-to-image belongs to multimodal generation—both differ significantly from the vision tasks used for training. The searcher still works effectively, demonstrating that NCode's discrete encoding scheme is general enough to map design choices from different modalities into the same representation space.

According to the original paper, the framework achieves state-of-the-art performance on multiple benchmarks, validating its generalization capability across tasks, modalities, and joint search space optimization.

Scale Validation on Billion-Parameter Diffusion Transformers

Scalability is key to assessing the practical value of an AutoML framework. The study applied the LLM searcher to large-scale training and joint optimization of billion-parameter Diffusion Transformers, consuming 896 GPU-days and ultimately achieving a best score of 0.91 on the GenEval benchmark.

The significance of this result: the searcher can handle not only small-scale benchmark tasks but also make effective configuration decisions in real large-scale training scenarios. The 896 GPU-days search cost itself also suggests that the value of a universal searcher needs to be measured within a benefit framework of "replacing manual tuning" and "finding better configurations."

GenEval is a benchmark for evaluating the instruction-following capability of text-to-image generation models. It decomposes "whether generation matches the text description" into fine-grained subtasks such as object count, color, spatial relationships, and attribute binding, scoring with automated detectors rather than relying on subjective human evaluation. Therefore, GenEval scores can objectively reflect how faithfully a model follows complex prompts. The 0.91 score in the article is the best result achieved on this benchmark, indicating that configurations obtained through AutoML joint optimization improve both generation quality and instruction following. Understanding this metric helps judge that the benefits of the searcher are not only in training efficiency but also in the substantive quality of downstream generation tasks.

Interpretability Through Reinforcement Learning and Chain-of-Thought

AutoML systems are often criticized as black boxes. This study injects Chain-of-Thought (CoT) reasoning capability into the searcher through reinforcement learning, providing a systematic way to interpret the internal decision-making mechanisms of the LLM searcher.

Analysis reveals a clear strategy transfer trajectory: the early stage is dominated by diversity-driven exploration, transitioning to anchor exploitation in later stages. This shift from exploration to exploitation aligns with general intuition about optimization search, but previously lacked evidence of being explicitly observed and explained within an LLM searcher. Interpretability not only enhances trust but also provides a handle for subsequent improvements to search strategies.

Chain-of-Thought (CoT) refers to having language models output intermediate reasoning steps before reaching conclusions, thereby improving performance on complex tasks and exposing the basis for decisions. In the AutoML context, combining CoT with reinforcement learning means that each time the searcher proposes a configuration suggestion, it first generates reasoning text about "why choose this" and then ranks candidate solutions accordingly. Reinforcement learning then optimizes these reasoning and selection strategies through final performance feedback, enabling the model to gradually learn more effective search logic. The observed trajectory of "early diversity exploration, later anchor exploitation" was identified from these reasoning texts, which is precisely what distinguishes this method from traditional black-box AutoML.

Implications for Open-World Machine Learning

The framework points toward a scalable, generalizable, and interpretable path for universal machine learning optimization. Its key contributions can be summarized in three points:

• Unified encoding: NCode unifies heterogeneous design choices—architecture, hyperparameters, data mixture—into discrete sequences, enabling joint search.
• Zero fine-tuning transfer: An LLM searcher trained on single-domain trajectories works directly across tasks and modalities.
• Interpretable decisions: Reinforcement learning plus chain-of-thought makes the searcher's optimization strategy observable.

Of course, the method remains at the research stage. The 896 GPU-days search cost, dependence on LLM reasoning capability, and performance boundaries on larger-scale tasks are all questions that need answering before production deployment. But directionally, it offers a compelling universal solution to AutoML's fragmentation problem.

One-line Summary

LLM as universal searcher with unified encoding enables cross-task, cross-modality joint AutoML optimization.

Paragraph Summary

This study proposes the NCode framework, using large language models as universal searchers that transform heterogeneous design choices—model architecture, training hyperparameters, and data mixtures—into discrete sequences through unified numerical encoding, reframing AutoML optimization as a unified ranking task and achieving joint optimization of all three for the first time. Trained only on optimization trajectories from a single vision task, the searcher reaches SOTA on cross-task, cross-modality benchmarks including image classification, image segmentation, speaker verification, and text-to-image, without task-specific fine-tuning. The study also completes large-scale validation on billion-parameter Diffusion Transformers, consuming 896 GPU-days and scoring 0.91 on GenEval. By injecting chain-of-thought reasoning through reinforcement learning, the authors reveal a trajectory from early diversity exploration to later anchor exploitation, offering a scalable, generalizable, and interpretable universal solution to AutoML's fragmentation problem.