OpenDiscoveryTrace: How 558 Trajectory Records Audit AI Scientist Reasoning

Output-Only Evaluation of AI Scientists Is Breaking Down

In recent years, Autonomous AI Scientists have become a major research focus — systems that can automatically generate hypotheses, write experimental code, and even draft papers. Yet evaluating these systems has always been a thorny problem.

Today's mainstream benchmarks almost exclusively focus on final outputs: Does the generated code run? Are the hypotheses reasonable? How good is the paper? This approach seems straightforward, but it has a fatal blind spot — it completely discards the reasoning process that led to those results.

This creates three unavoidable problems: there's no way to audit whether the scientific methodology was sound, no way to diagnose the specific cause of failures, and no way to distinguish "systematic reasoning" from "lucky guessing." A model might arrive at the correct answer through a valid chain of logic, or it might simply get lucky — but from the final output alone, the two are indistinguishable.

The new paper OpenDiscoveryTrace is a direct response to this pain point: it argues that evaluation shouldn't just look at what a model produces, but at how it reasons.

Autonomous AI Scientists are a class of agent systems capable of completing the full scientific research loop with minimal human intervention. A typical workflow includes: retrieving and interpreting literature, proposing verifiable hypotheses, automatically designing and executing computational experiments (via code interpreters, external APIs, or databases), analyzing results, and finally generating structured reports or paper drafts. Projects like AI Scientist (Sakana AI, 2024) have drawn widespread attention to this space in recent years, but evaluation challenges have also come to the fore — scientific tasks often have no single correct answer, and output quality is difficult to quantify with traditional accuracy metrics, making output-only evaluation even more fragile.

558 Complete Reasoning Traces: The OpenDiscoveryTrace Dataset Explained

OpenDiscoveryTrace is a public dataset containing 558 complete AI scientist agent trajectories, with its core innovation being the structured recording of the reasoning process.

Nine-Field Trace Structure: Reconstructing AI's Mental Process Step by Step

Each trajectory in the dataset consists of multiple execution steps, and every step is broken down into nine fields:

• Thoughts: The model's reasoning content
• Tool Calls: Which external tools were invoked
• Observations: Results returned by the tools
• Errors: Problems encountered during execution
• Revision Triggers: When and why the model adjusted its strategy
• Self-reported Confidence: The model's confidence in its own judgments

This fine-grained recording approach allows researchers to replay the complete mental trajectory of an AI solving a problem step by step — rather than seeing only an isolated endpoint.

"Trace" is a key concept in agent research, referring to the ordered record of all intermediate states and actions an agent takes while completing a task. It is essentially a complete log of the "think–act–observe" loop (the Reason-Act cycle described by the ReAct framework). Unlike traditional logs that only save the final answer, a trace preserves where the model branched, where it backtracked, which tools it called, and what those tools returned — all intermediate information that forms the basis for post-hoc diagnosis and interpretability analysis. OpenDiscoveryTrace further refines traces into nine fields, with "Revision Triggers" and "Self-reported Confidence" being particularly uncommon in existing public datasets, providing a new observational window for studying model metacognitive behavior.

124 Tasks Spanning Four Scientific Domains

These traces come from models executing 124 scientific tasks across four real-world research scenarios: drug discovery, materials science, genomics, and scientific literature analysis. This domain coverage ensures the evaluation isn't limited to toy problems, but instead reflects genuine scientific workflows.

Seven Models, Side by Side: A Comprehensive Comparison of Frontier Closed-Source and Open-Weight Models

The dataset strikes a careful balance in model selection, covering two categories totaling seven models:

Three frontier models: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro, each contributing 124 traces, with fully balanced distribution across domains and difficulty levels.

Four open-weight models: Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B, each with 30 traces.

An additional 60 "live retrieval" variant traces are included to examine model behavior in scenarios where information is fetched from the web. This comparison design between frontier closed-source and open-weight models provides a valuable foundation for studying reasoning differences across models of varying scale and provenance.

Process Traces Reveal Critical Differences That Output Evaluation Cannot See

The paper's most striking findings come from a preliminary analysis of 363 LLM-judge-evaluated traces — process traces expose behavioral differences that pure output evaluation is completely unable to capture.

Similar Success Rates, 30x Difference in Error Rates

The three frontier models had very similar task success rates, all falling in the 84%–89% range. Looking at this number alone, one would conclude they are roughly equivalent.

But process analysis tells a completely different story: Claude Opus 4.6 produced 30 times more errors than GPT-5.4 (an average of 2.5 errors per trace vs. 0.08). This difference is highly statistically significant (p < 0.0001, Cliff's δ = 0.613) and is far from coincidental.

Cliff's δ is a non-parametric effect size statistic that measures the practical magnitude of the difference between two groups, with values ranging from -1 to 1. Unlike p-values, which only reflect whether a difference reaches statistical significance, effect size answers the question of how large the difference is. δ = 0.613 qualifies as a "large effect" under standard criteria (typically |δ| > 0.474), meaning that regardless of sample size, the difference in error frequency between Claude Opus 4.6 and GPT-5.4 is substantively meaningful. Reporting both p-values and effect sizes is current best practice in statistical analysis, and effectively prevents small differences from being incorrectly amplified into "important findings" in large-sample settings.

Tool Misuse vs. Reasoning Errors: Fundamentally Different Failure Modes

An even deeper insight lies in the nature of the errors:

• In Claude Opus 4.6's errors, 66.7% were tool misuse
• In GPT-5.4's errors, 83.6% were reasoning errors

This means that even when two models achieve similar success rates, the root causes of their failures are fundamentally different — one "uses the wrong tool," the other "thinks the wrong logic." This kind of information is critical for understanding a model's true capability boundaries and guiding targeted improvements — yet all of it is hidden from output-only evaluation.

Five Benchmark Tasks and an Open CC BY 4.0 Ecosystem

To make the dataset genuinely usable for research, the authors define five benchmark tasks and provide baseline results across multiple model types ranging from logistic regression and random forests to LSTMs and Transformers, making it easy for future researchers to compare against.

Equally commendable is its openness: the dataset, trace schema, agent harness, and benchmark definitions are all released under a CC BY 4.0 license. This removes barriers for research in process-level evaluation, scientific agent auditing, and AI governance.

CC BY 4.0 (Creative Commons Attribution 4.0 International) is one of the most widely used open licenses in academic datasets. It allows anyone to freely use, modify, and redistribute data, with the only requirement being attribution to the original source. Compared to more restrictive licenses like CC BY-NC (no commercial use) or CC BY-SA (share-alike required), CC BY 4.0 places no restrictions on commercial applications or closed-source downstream products, giving it maximum accessibility for both industry and academia. Simultaneously releasing the dataset, evaluation framework, and benchmark definitions also aligns with the principles of Reproducible Science, enabling future researchers to replicate and extend experiments under identical conditions.

From "Output-Oriented" to "Process-Auditable": A Paradigm Shift in Evaluation

The value of OpenDiscoveryTrace lies not only in providing a dataset, but in advancing a shift in evaluation paradigm.

As AI plays an increasingly important role in scientific research, "whether it can reach the right conclusion" is no longer sufficient — we need to know "how it reached that conclusion." An auditable reasoning process is a prerequisite for building trust, discovering systematic flaws, and enabling AI governance.

For developers, process traces help identify whether a model is a "tool-user" or a "reasoner," enabling targeted improvements. For the scientific community, they provide a means of verifying the methodological rigor of AI-generated discoveries. For regulators and governance bodies, they offer a concrete, actionable audit framework.

As the capabilities of autonomous AI scientists continue to grow, process-level evaluation will inevitably become an indispensable component — and OpenDiscoveryTrace is a foundational piece of work in precisely that direction.