SEER-Repro
A benchmark dataset for evaluating the scientific reproduction capability of LLM agents. It contains 20 published papers based on the SEER cancer database together with the data required for their reproduction, and decomposes paper conclusions into 2,190 atomic claims as quantifiable evaluation units.
Timeline (last 90 days)
In the SEER-Repro experiment, the researchers evaluated 2,190 atomic claims across 20 empirical studies.
The agent framework achieved a 77.4% end-to-end autonomous reproduction rate in the SEER-Repro experiment.
The agent framework identified analytical fragility in 9.5% of the reproducible findings.
The study constructed the SEER-Repro benchmark, comprising 20 published papers based on the SEER database and the data and resources needed for reproduction.
The SEER-Repro benchmark decomposes each paper's conclusions into atomic claims, yielding 2,190 atomic claims in total.
The developed AI agent successfully reproduced 77.4% of the atomic claims on the SEER-Repro benchmark.
All Facts (6)
In the SEER-Repro experiment, the researchers evaluated 2,190 atomic claims across 20 empirical studies.
50%UnverifiedThe agent framework achieved a 77.4% end-to-end autonomous reproduction rate in the SEER-Repro experiment.
50%UnverifiedThe agent framework identified analytical fragility in 9.5% of the reproducible findings.
50%UnverifiedThe SEER-Repro benchmark decomposes each paper's conclusions into atomic claims, yielding 2,190 atomic claims in total.
50%UnverifiedThe developed AI agent successfully reproduced 77.4% of the atomic claims on the SEER-Repro benchmark.
50%UnverifiedThe study constructed the SEER-Repro benchmark, comprising 20 published papers based on the SEER database and the data and resources needed for reproduction.
50%