NatureBench
A benchmark dataset for evaluating the performance of AI research agents on natural science tasks, measuring the proportion of results that match or exceed the state of the art (SOTA).
Timeline (last 90 days)
The SwarmRSI framework comprehensively sets new SOTA across 118 tasks on the three benchmarks OpenProblems, NatureBench and MLEBench Lite.
On NatureBench tasks, removing the H mutation reduces the proportion matching SOTA by 10 percentage points, and removing the P mutation reduces it by 13.3%.
With DeepSeek V4 Pro as the backbone, SwarmRSI's proportion matching SOTA on NatureBench is 21.1 percentage points higher than Claude Code with the same backbone.
Among the 90 tasks in NatureBench, 77% of the final best programs came from at least one recombination, crossover or feedback-triggered update.
In cross-task transfer experiments, the research states inherited from three source tasks raised the average base value of 12 held-out NatureBench tasks from 12.01 under cold start to 21.89, and the proportion matching SOTA jumped from 41.7% to 86.1%.
All Facts (5)
The SwarmRSI framework comprehensively sets new SOTA across 118 tasks on the three benchmarks OpenProblems, NatureBench and MLEBench Lite.
50%UnverifiedOn NatureBench tasks, removing the H mutation reduces the proportion matching SOTA by 10 percentage points, and removing the P mutation reduces it by 13.3%.
50%UnverifiedWith DeepSeek V4 Pro as the backbone, SwarmRSI's proportion matching SOTA on NatureBench is 21.1 percentage points higher than Claude Code with the same backbone.
50%UnverifiedAmong the 90 tasks in NatureBench, 77% of the final best programs came from at least one recombination, crossover or feedback-triggered update.
50%UnverifiedIn cross-task transfer experiments, the research states inherited from three source tasks raised the average base value of 12 held-out NatureBench tasks from 12.01 under cold start to 21.89, and the proportion matching SOTA jumped from 41.7% to 86.1%.
50%