Scientist World: Building a Runnable Research World for AI Scientists

Title

Scientist World: Building a Runnable Research World for AI Scientists

Meta Description

Scientist World turns every paper into an executable research environment, with the first release of 1,000 validated environments. Frontier AI scientists outperform human researchers in 74% of cases, 6,400 hours of research trajectories are used for training, and a small model's research ideation ability improves by 158%.

Article Content

AI scientists have already emerged, but the world in which they are meant to grow has not yet been built. Human scientists take this for granted—open problems and shared ideas accumulated over centuries sit quietly in the scientific literature. For AI scientists, that world is readable but not runnable. To address this, we built Scientist World: every research paper becomes an executable environment.

From Papers to Executable Environments

The core idea of Scientist World is not complicated, yet it is quite ambitious. The research team designed an autonomous pipeline that automatically extracts open research questions from each paper and reproduces the paper's proposed method as a baseline. When an AI scientist enters the environment, its performance is evaluated against the results achieved by the human authors. This effectively transforms a static literature repository into a dynamic experimental arena.

The first release includes 1,000 validated environments. Each environment has been validated through reproduction to ensure that the baseline the AI scientist faces is reliable and runnable. This is crucial: if the baseline itself is wrong, all subsequent evaluations lose their meaning.

Using this world as an evaluation benchmark, the team found that frontier AI scientists can already outperform human researchers in 74% of cases. This number means that, at least on some validated research problems, AI can not only understand the paper but also propose solutions superior to the original.

Executable Environment is the core concept of this article, referring to a programmatic sandbox capable of running code, receiving inputs, and producing observable outputs, rather than a static document meant only for reading. Turning a paper into an executable environment means automatically extracting the method description, hyperparameter settings, and dataset information from the paper, reconstructing its code implementation, and configuring the dependencies required to run it. This differs from simple code reproduction: it also requires the environment to encapsulate open problems so that AI can propose hypotheses, run experiments, and receive feedback. The key value of an executable environment lies in providing a verifiable comparison baseline: any attempt by AI can be directly compared with the results reported in the original paper, thereby determining whether it has truly advanced the problem. Such environments also require certain fault-tolerance and isolation mechanisms to prevent experimental errors from affecting the overall system.

Training Ground: 6,400 Hours of Research Trajectories

Scientist World has another identity: a recorder. It contains 6,400 hours of research trajectories, where every idea, experiment, failure, and success is recorded. These trajectories form a raw archive of how AI does science and also provide material for subsequent training.

As a training ground, the team used this data to improve a small model's research ideation ability by 158%, enabling it to match frontier models. The magnitude of the improvement shows that research ideation does not have to rely solely on stacking model scale; high-quality process data can also bring significant change.

Research Trajectory refers to the complete sequence of operations produced by an AI scientist while solving a research problem, including proposed ideas, designed experiments, executed code, observed results, and adjustments after failure. Unlike final answers, trajectories record the intermediate process of reasoning and trial and error, and therefore contain richer learning signals. In training, trajectories can be used for process supervision or imitation learning, allowing the model to learn how to decompose problems, choose experimental strategies, and recover from failure. The scale of 6,400 hours of trajectories enables researchers to analyze patterns in AI scientific behavior and to filter high-quality trajectories for fine-tuning. The quality of trajectory data is more critical than quantity; repetitive, invalid, or overly simple attempts need to be filtered out, otherwise noise may be introduced.

Three Layers of Value: Evaluation, Recording, and Training

The architecture of Scientist World can be divided into three layers. The evaluation layer provides runnable paper environments, using human results as a reference frame; the recording layer accumulates research trajectories, making the AI's scientific process traceable; and the training layer uses these trajectories to improve the model's research ideation ability. The three layers reinforce one another: evaluation produces records, records are used for training, and the trained model returns to evaluation for testing.

This closed-loop design addresses a neglected problem. Previous evaluations of AI scientific research mostly stayed at the level of question answering or code completion, lacking a real scientific research process. Scientist World attempts to let AI scientists grow in an environment close to real research, rather than merely acting as readers of literature.

Closed-loop Design in this article specifically refers to a system architecture in which evaluation, recording, and training are interconnected and self-reinforcing. Research trajectories generated during evaluation are accumulated by the recording layer, the training layer uses these trajectories to improve the model, and the improved model returns to the evaluation environment for testing, forming a cycle. This design draws on the idea of environment interaction in reinforcement learning, but places greater emphasis on the complexity of real scientific research processes. The key challenge of the closed loop is to avoid evaluation contamination: if training data and evaluation environments overlap too much, the model may memorize answers rather than learn to do research. Therefore, trajectories used for training and environments used for evaluation must be strictly separated, and the evaluation set must be updated regularly. A good closed loop should be able to continuously generate new open problems and high-quality trajectories, driving a spiral of capability improvement.

Significance and Open Questions

The AI scientist's own world has arrived; it grows out of science and will grow along with science. However, the 74% surpass rate needs to be understood in context: it measures performance in validated environments, not the ability to independently conduct research on fully open frontier problems. The quality distribution of the 6,400 hours of trajectories, the degree of automation in the reproduction process, and whether the improvement of small models can generalize to larger scales are all directions worth observing in future work.

Regardless, the act of turning papers into runnable environments provides a foundation for evaluating and training AI scientific research capabilities that is closer to real research.

One-line Summary

Scientist World turns papers into executable environments to evaluate and train AI scientists.

Paragraph Summary

Scientist World is a system that automatically transforms research papers into executable environments, with an initial release of 1,000 reproduction-validated environments. Solutions proposed by AI scientists in these environments can be directly compared with the results of human authors, and frontier models outperform humans in 74% of cases. The system also records 6,400 hours of research trajectories for training small models, improving their research ideation ability by 158% and enabling them to match frontier models. Overall, it forms a closed loop of evaluation, recording, and training, allowing AI to grow in an environment close to real scientific research processes rather than merely reading literature. The article also points out the specific context of the surpass rate and open questions such as trajectory quality.