TRACES Benchmark
A benchmark for evaluating whether large language models can identify problematic papers (retracted, fraudulent, or pseudoscientific) and refuse to design experiments based on them, testing model behavior through probe questions embedded in paper passages with reputable control papers.
Timeline (last 90 days)
The test was initiated by Case Western Reserve University organic chemist Valentin Rodionov, and the results were published on arXiv as the TRACES benchmark.
In the TRACES test, every model had a failure rate above 71% on the 'agentic' probes; 22 of the 30 models had failure rates above 90%.
The paper's authors argue that the models' refusal behavior is concentrated on a few high-profile topics and specific probes, and disappears in structurally matched control groups, which indicates that the model behavior is 'topic-triggered safety behavior' rather than genuine cognitive reasoning ability.
The TRACES benchmark's control design includes 42 problematic papers and control papers that are structurally topic-matched but have good academic reputations; if the models' refusal rate for problematic papers drops significantly in the control group, it indicates that their behavior is driven by topic prominence rather than methodological judgment.
The TRACES benchmark was released as an arXiv preprint and has not yet undergone peer review.
All Facts (5)
The test was initiated by Case Western Reserve University organic chemist Valentin Rodionov, and the results were published on arXiv as the TRACES benchmark.
50%UnverifiedIn the TRACES test, every model had a failure rate above 71% on the 'agentic' probes; 22 of the 30 models had failure rates above 90%.
50%UnverifiedThe paper's authors argue that the models' refusal behavior is concentrated on a few high-profile topics and specific probes, and disappears in structurally matched control groups, which indicates that the model behavior is 'topic-triggered safety behavior' rather than genuine cognitive reasoning ability.
50%UnverifiedThe TRACES benchmark's control design includes 42 problematic papers and control papers that are structurally topic-matched but have good academic reputations; if the models' refusal rate for problematic papers drops significantly in the control group, it indicates that their behavior is driven by topic prominence rather than methodological judgment.
50%UnverifiedThe TRACES benchmark was released as an arXiv preprint and has not yet undergone peer review.
50%