SciResearch-Eval
The first interactive benchmark for evaluating the scientific judgment of AI agents, testing their calibration and scientific taste by having agents autonomously design experiments, use potentially flawed simulators, and provide 95% confidence intervals.
Timeline (last 90 days)
SciResearch-Eval is the first benchmark (according to its authors) that simultaneously combines agent-autonomous experiment design, a simulator with deliberately injected flaws, and scoring of calibration intervals against exact ground truth.
SciResearch-Eval contains 93 held-out tasks, 9 models, and 2,511 episodes in total.
In SciResearch-Eval, most models' 95% confidence intervals covered the true value in only 20% to 62% of cases.
GPT-6 Astra's 95% confidence interval coverage reached 94%.
Claude Fable 5.2's 95% confidence interval coverage reached as high as 99%.
At default reasoning effort, Claude Opus noticed the missing physics terms in the simulator in 35% of episodes.
At default reasoning effort, GPT-6 Astra noticed the missing physics terms in the simulator in 66% of episodes.
Requiring models to state their own strategy improved interval scores by 1.6 to 2.5 times, and this result was reproduced on a pre-registered, entirely new data split.
Although strategy statements improved interval scores, the coverage of models' 95% confidence intervals remained almost unchanged.
The research team has released the SciResearch-Eval environment, scorer, and execution framework.
All Facts (10)
SciResearch-Eval is the first benchmark (according to its authors) that simultaneously combines agent-autonomous experiment design, a simulator with deliberately injected flaws, and scoring of calibration intervals against exact ground truth.
50%UnverifiedSciResearch-Eval contains 93 held-out tasks, 9 models, and 2,511 episodes in total.
50%UnverifiedIn SciResearch-Eval, most models' 95% confidence intervals covered the true value in only 20% to 62% of cases.
50%UnverifiedGPT-6 Astra's 95% confidence interval coverage reached 94%.
50%UnverifiedClaude Fable 5.2's 95% confidence interval coverage reached as high as 99%.
50%UnverifiedAt default reasoning effort, Claude Opus noticed the missing physics terms in the simulator in 35% of episodes.
50%UnverifiedAt default reasoning effort, GPT-6 Astra noticed the missing physics terms in the simulator in 66% of episodes.
50%UnverifiedRequiring models to state their own strategy improved interval scores by 1.6 to 2.5 times, and this result was reproduced on a pre-registered, entirely new data split.
50%UnverifiedAlthough strategy statements improved interval scores, the coverage of models' 95% confidence intervals remained almost unchanged.
50%UnverifiedThe research team has released the SciResearch-Eval environment, scorer, and execution framework.
50%