Testing a Research Automation Agent: Zero-Shot Reproduction of a NeurIPS Paper End-to-End

When Research Reproduction Enters the Fully Automated Era

Reproducing papers has always been one of the most time-consuming and patience-testing parts of research work. From searching literature and understanding methods, to setting up environments, getting code to run, and validating results, researchers often need to invest days or even weeks of effort. Now, a research automation agent called Klaus Goh is attempting to compress this tedious loop into a single "one-click delivery."

According to a hands-on demonstration by a Bilibili content creator, Klaus Goh is positioned as "the world's first fully automated research reproduction agent," capable of independently completing the entire chain of research reproduction: paper searching, environment setup, code implementation, experiment validation, and result analysis, ultimately delivering a complete reproduction report along with reproducible experimental results. The subject of this test was a cutting-edge time series forecasting work published by IBM at NeurIPS—TTM (Tiny Time Mixers).

The Reproduction Goal: A Tiny Model Challenging Large Models

The Core Idea of the TTM Paper

TTM is a rather disruptive achievement in the field of time series forecasting. Its core idea is to use an extremely small MLP Mixer architecture for zero-shot time series forecasting.

The origin and evolution of the MLP Mixer architecture: MLP Mixer was originally proposed by Google Research in 2021 for image recognition tasks, with the paper published at NeurIPS 2021. Its core innovation lies in completely abandoning the self-attention mechanism in Transformers, which has a computational complexity of O(n²)—self-attention requires every position in the sequence to perform pairwise interaction computations with all other positions, causing the computation to grow quadratically with sequence length, resulting in considerable resource consumption in long-sequence scenarios. MLP Mixer instead captures the global and local relationships of data through two types of lightweight fully connected layers (MLPs): one MLP mixes features along the "channel" dimension, capturing correlations between different feature dimensions; the other MLP mixes features along the "position" dimension, capturing dependencies between different time steps in the sequence. This "bidirectional mixing" design reduces computational complexity to linear, greatly lowering the resource consumption of training and inference. TTM transfers this idea to the time series domain, fully leveraging the structural patterns that naturally exist in time series data along both the channel (multivariate) dimension and the temporal dimension—such as correlations between different sensors and the periodicity of time series data—thereby effectively modeling complex temporal patterns while keeping the parameter count extremely low, achieving dual optimization of parameter efficiency and forecasting performance. It's worth noting that the success of MLP Mixer in the image domain has already demonstrated that the attention mechanism is not the only effective inductive bias in deep learning; "replacing complex nonlinear interactions with simpler linear operations" works equally well on certain structured data, and TTM is precisely an extended validation of this design philosophy in the time series domain.

In today's ever-intensifying large model arms race, the time series forecasting field is no exception—models like TimesFM and PatchTST routinely have billions of parameters, demanding extremely high computational power. TTM, however, goes against the trend, with only about 0.8M (million-level) parameters and a file size of just three megabytes. It doesn't even require a GPU and can run smoothly on an ordinary CPU.

The technical meaning and value of zero-shot forecasting: Zero-shot Forecasting refers to a model performing inference directly using pretrained weights on a target dataset it has never seen, without any fine-tuning on the target domain. The core of this capability relies on thorough learning of massive and diverse time series data during the pretraining stage—TTM's pretraining corpus covers time series datasets from multiple domains including energy, transportation, meteorology, and finance, enabling the model to learn cross-domain general representations of temporal patterns. This process is highly similar to the "pretrain-finetune" paradigm of large language models: through pretraining on massive heterogeneous data, the model forms a general encoding capability for underlying patterns, then generalizes to specific downstream tasks—the only difference being that the zero-shot scenario skips the latter fine-tuning step. In the time series forecasting domain, zero-shot capability is particularly precious: in many real-world application scenarios such as industrial IoT and medical monitoring, labeled data in the target domain is extremely scarce, or the data distribution continuously drifts due to factors like seasonality and equipment aging, making it difficult for traditional supervised learning methods to generalize. TTM's ability to achieve competitive zero-shot forecasting performance with an ultra-small parameter count of 0.8M means that its pretraining strategy has reached extremely high parameter efficiency in the compressed encoding and cross-domain generalization of temporal patterns, and it provides strong empirical support for the subsequent research direction of "Small Foundation Models."

This "small and beautiful" design directly points to a highly valuable application scenario: time series forecasting on edge devices. When a model can be deployed on resource-constrained terminals, its practical deployment potential is greatly expanded.

The Full Reproduction Process by Klaus Goh

Step 1: From Paper to Environment, Precisely Locking onto Key Information

Upon receiving the task of reproducing TTM, the first capability Klaus Goh demonstrated was rapid localization of the paper's key information. After reading the paper, it immediately captured three core elements: the model architecture, the pretrained weights, and the evaluation benchmark, and instantly locked onto the official repository on Hugging Face.

The technical architecture and evolutionary path of research automation agents: Agents like Klaus Goh are typically built on the tool use and task planning capabilities of large language models (LLMs), combined with external tool interfaces such as code execution sandboxes (like Jupyter kernels or Docker containers), file system access, academic search via arXiv/Semantic Scholar, and GitHub/Hugging Face APIs, forming a complete loop of "perception (reading papers and code) → planning (decomposing reproduction steps) → execution (calling tools to run code) → feedback (analyzing errors and correcting)." Unlike early rule-based RPA (Robotic Process Automation), LLM-driven agents can understand unstructured paper text, autonomously decompose the high-level goal of "reproducing the TTM paper" into a sequence of concrete executable subtasks, and dynamically adjust strategies when encountering environment errors or dependency conflicts. This dynamic fault-tolerance capability comes from the LLM's understanding of code semantics and error messages, rather than preset exception-handling rules—essentially transforming "debugging," a cognitive task that originally relied on expert experience, into a standardized process that the LLM can autonomously reason through. Its implementation approach is similar to systems successively proposed by academia such as AI Scientist (Sakana AI), SWE-Agent (Princeton), and AutoML-Agent, but it focuses more on the specific domain of research reproduction, representing the frontier exploration of AI agents evolving from "conversational assistants" toward "embodied autonomous executors."

This step is actually where reproduction work is most prone to pitfalls—using the wrong weights, the wrong benchmark, or having dependency conflicts can all render subsequent experiments useless. The agent's ability to automatically locate official resources and set up the environment means it already possesses a practically usable understanding of the connections within the research ecosystem (papers, code repositories, model hosting platforms).

Step 2: Data Processing and Zero-Shot Inference

Once the environment was ready, Klaus Goh completed everything automatically—from loading and standardizing the ETT-1 dataset to loading the pretrained model. It then launched zero-shot inference.

Test data showed that zero-shot inference took only 10 seconds to compute over 2,700 prediction points. This speed is partly due to the lightweight design of the TTM model itself, and it also confirms that the paper's claim of "CPU-runnable" is no empty boast.

Step 3: Result Validation—A Small Model Outperforms Large Models

The most critical part is result validation. The conclusion of the hands-on reproduction was consistent with the paper's core claim: TTM achieved an MSE of only 0.363 on the ETT-1 dataset, surpassing a range of large models such as TimesFM and PatchTST, while its parameter count is only 0.8M.

The industry background of the ETT dataset and evaluation metrics: The ETT (Electricity Transformer Temperature) dataset is one of the most widely used open benchmarks in the time series forecasting field, released by Huazhong University of Science and Technology in 2021 alongside the Informer paper (AAAI 2021 Best Paper). This dataset records the oil temperature and six power load channels from two county-level power stations in a region of China. Its multivariate, long-sequence characteristics make it a standard testbed for examining a model's ability to model long-term dependencies. ETTh1 is sampled at hourly granularity, containing approximately 17,420 time steps, covering about two years of continuous records, and is divided into training, validation, and test sets. The evaluation protocol is highly unified in academia, ensuring direct comparability between results from different papers. MSE (Mean Squared Error) is a core metric for measuring prediction accuracy, computing the mean of the squared differences between predicted and actual values—the lower the value, the more accurate the prediction. It's worth noting that MSE penalizes large errors quadratically, making it particularly sensitive in industrial data scenarios with many outliers, so TTM's ability to maintain an advantage on this metric demonstrates its excellent prediction robustness. TTM's 0.363 MSE is significantly lower than the performance of TimesFM (released by Google, with about 200 million parameters, roughly 250 times that of TTM) and PatchTST (a Transformer-based time-series-specific model) under the same ETTh1 evaluation protocol. After controlling for evaluation conditions, this constitutes a performance surpass with reference value, and it holds important significance for the industry's assessment of the applicability boundary of the "Scaling Law" (the relationship between model scale and performance) in the time series forecasting domain—it shows that when data has a sufficiently strong intrinsic structure, a carefully designed inductive bias can partially replace the fitting capability brought by parameter scale.

In other words, a three-megabyte model defeated opponents with parameter counts several orders of magnitude larger on standard evaluation benchmarks. If this conclusion can be widely reproduced, it will have a substantial impact on the choice of technical roadmaps in the time series forecasting field—edge-side deployment would no longer be a compromise under computational constraints, but potentially a superior engineering solution.

The Deliverable: A Complete Reproduction Report

After the entire reproduction loop was completed, Klaus Goh delivered a fairly complete set of results:

• A complete reproduction report
• All runnable Python code
• Zero-shot forecasting comparison charts
• Error metric comparison charts
• Multi-channel visualization charts
• The original paper, directly readable

From paper searching and getting experiments to run, to report output and chart synthesis, the entire process was completed independently by the agent. This is no longer just "running code"—it simulates the complete workflow of a research assistant from reading a paper to writing a report.

A Note of Sober Observation

Worth mentioning: this article is based on a single-source hands-on demonstration, showing a reproduction task with "a certain result and a clear path"—TTM has an official open-source repository, a public dataset, and a clear evaluation benchmark, making it relatively "friendly" for an agent.

The real challenges of research reproduction often arise when papers lack details, code isn't open-sourced, and hyperparameters require repeated tuning. Whether research automation agents can remain equally robust in these "non-ideal" scenarios still awaits testing with more complex cases. This limitation is in line with the general bottleneck in the current AI agent field: in scenarios where the task space is highly constrained, the goal is clear, and feedback signals are clear, agents perform excellently; but when task boundaries are blurry and open-ended exploration is required, their planning capabilities and fault-tolerance mechanisms still face significant challenges.

Nevertheless, Klaus Goh's hands-on test still reveals an important trend: research automation is evolving from an "auxiliary tool" toward an "independent executor." When AI can complete the end-to-end reproduction of a top-tier conference paper, the role of researchers may gradually shift from "hands-on implementation" to "raising questions and judging value." This may be the more profound transformation that AI research automation brings.