[KongchangAI]
Benchmark

SciResearch-Eval

The first interactive benchmark for evaluating the scientific judgment of AI agents, testing their calibration and scientific taste by having agents autonomously design experiments, use potentially flawed simulators, and provide 95% confidence intervals.

Timeline (last 90 days)

Nov 17

SciResearch-Eval is the first benchmark (according to its authors) that simultaneously combines agent-autonomous experiment design, a simulator with deliberately injected flaws, and scoring of calibration intervals against exact ground truth.

Unverified50%
Nov 17

SciResearch-Eval contains 93 held-out tasks, 9 models, and 2,511 episodes in total.

Unverified50%
Nov 17

In SciResearch-Eval, most models' 95% confidence intervals covered the true value in only 20% to 62% of cases.

Unverified50%
Nov 17

GPT-6 Astra's 95% confidence interval coverage reached 94%.

Unverified50%
Nov 17

Claude Fable 5.2's 95% confidence interval coverage reached as high as 99%.

Unverified50%
Nov 17

At default reasoning effort, Claude Opus noticed the missing physics terms in the simulator in 35% of episodes.

Unverified50%
Nov 17

At default reasoning effort, GPT-6 Astra noticed the missing physics terms in the simulator in 66% of episodes.

Unverified50%
Nov 17

Requiring models to state their own strategy improved interval scores by 1.6 to 2.5 times, and this result was reproduced on a pre-registered, entirely new data split.

Unverified50%
Nov 17

Although strategy statements improved interval scores, the coverage of models' 95% confidence intervals remained almost unchanged.

Unverified50%
Nov 17

The research team has released the SciResearch-Eval environment, scorer, and execution framework.

Unverified50%

All Facts (10)

Source Articles