The Digital Trap of AI-Written Papers: Provenance Auditing Matters More Than Fabrication Detection

AI research agents can now independently write complete academic papers, but a new analysis reveals a overlooked problem: most of the numerical errors in these papers are not deliberate fabrication, but rather "misattribution"—citing values from the wrong experiment, wrong metric, or wrong run batch within their own experimental records.

Re-examining 540 AI Paper Judgments: Failures Center on "Alignment" Rather Than "Fabrication"

A recent public benchmark for AI research agents covered nine agents and fifteen open research questions, and released all 540 judgment results. Researchers re-analyzed these results and manually annotated 256 numerical claims across 18 submissions.

The key finding: failures are concentrated in the "alignment" between the paper and the underlying work, rather than overt cheating. About four-fifths of unsupported claims actually correspond to values that exist in the agent's own files—only these values come from the wrong experiment, wrong metric, or wrong run batch. In other words, the AI did not fabricate numbers out of thin air; it "misremembered the source."

This conclusion challenges the simple binary used in previous benchmarks. Many integrity checks either only test the paper's internal consistency or only check whether a reported value has appeared in the agent's output. A single LLM judge's "fail" verdict often only indicates failure without explaining its nature.

Deterministic Checker: Calibrating Claim-Evidence Matching with "Numerical Decoys"

To address this issue, researchers introduced a deterministic "claim-to-evidence" checker. Its core idea: for each numerical claim, assess whether the surrounding context is sufficient to uniquely identify the claimed result, and calibrate the claim using "numerical decoys"—i.e., check whether the agent might have confused similar but incorrect values.

The checker's rules were fixed before annotation, avoiding post-hoc adjustments. In comparison with blind annotators, it certified claims with over 90% precision (78 out of 85 correct; Cohen's kappa = 0.89), but could only verify a subset of claims. Notably, most of the claims it could not verify were still traceable, so "unverifiable" does not equal "false."

This design philosophy embodies a "provenance-first" auditing approach: rather than trying to judge whether a number is "real," judge whether it can be traced to the correct source.

Cohen's kappa is a statistic measuring the degree of agreement between two annotators (or a system and a human annotator), accounting for the possibility of chance agreement. Values typically range from 0 to 1, where 0 indicates agreement equivalent to chance and 1 indicates perfect agreement. Generally, 0.61–0.80 is considered "substantial agreement" and above 0.81 "almost perfect agreement." The kappa=0.89 in this article indicates that the deterministic checker's judgments highly align with blind annotators—far from coincidental. This metric is widely used in evaluation studies requiring manual annotation because it reflects true agreement better than simple accuracy.

Numerical decoys are a technique for calibrating a checker's sensitivity: deliberately placing values similar to the target number but from incorrect sources (e.g., values from different experiments, different metrics, or different run batches) near the context of the claim to be verified, then observing whether the checker is misled by these "decoys." If the checker correctly rejects decoys and matches to the unique correct source, it demonstrates that it is truly evaluating the correspondence between context and evidence rather than merely performing string matching. This approach borrows from the idea of "mutation testing" in software testing—injecting known errors to assess detector robustness.

LLM Verifiers vs. Deterministic Checkers: Complementary, Not Replacements

Researchers also compared the performance of LLM verifiers working with the same evidence. LLM verifiers recovered more traceable claims, but they accepted about three times as many misattributed claims as the deterministic checker. This means LLMs are more "lenient" but also more easily fooled by surface similarity.

Both methods have shortcomings: the deterministic checker is precise but incomplete in coverage, while the LLM verifier has broader coverage but lower precision. Researchers suggest combining the two—using the deterministic checker for high-confidence certification, the LLM verifier to expand recall, and then manually or rule-based reviewing the claims accepted by the LLM.

Precision and recall are two core metrics for evaluating classification system performance. Precision refers to the proportion of claims certified as "correct" that are truly correct, reflecting the ability to avoid false positives; recall refers to the proportion of all truly correct claims that the system successfully certifies, reflecting the ability to avoid false negatives. There is usually a trade-off between the two: improving precision often sacrifices recall. In this article, the deterministic checker has over 90% precision but incomplete coverage (low recall), while the LLM verifier has higher recall but more false acceptances (low precision)—a typical manifestation of this trade-off. Combining both can pursue a better F1 score (the harmonic mean of precision and recall).

Application Results: Significant Differences in Traceability Across Agents

After applying the checker to 99 submissions, researchers found huge differences in traceability across AI agents. Some agents' numerical claims could almost all be traced to correct sources, while others frequently had attribution errors. This difference was completely obscured in simple "pass/fail" judgments.

Furthermore, when switching to an LLM judge from another vendor, the official agent rankings were preserved, but the number of passing papers doubled. This shows that a single judge's strictness significantly affects pass rates but does not necessarily change relative rankings. For a community that relies on benchmarks to evaluate AI research capabilities, this is an important methodological reminder.

"Pass rate" and "ranking" in benchmarks are two different levels of evaluation output. Pass rate is greatly influenced by judge strictness: switching to a more lenient LLM judge may double the number of passing papers, but as long as the judge is consistently lenient or strict across all agents, the ranking order among agents can still be preserved. This phenomenon is called the "judge effect" in evaluation research. It reminds researchers that absolute pass rates should not be directly compared across benchmarks, while rankings are more robust when judges change. However, preserved rankings do not mean the evaluation is valid—if the judge systematically ignores certain types of errors (such as misattribution), rankings may mask substantial differences in agent traceability.

Implications for AI Research Evaluation

The core contribution of this work is not to accuse AI agents of "fabrication," but to point out blind spots in current evaluation frameworks. If integrity checks only focus on "whether the number appears in the output," they will miss the deeper question of "whether the number comes from the correct experiment."

Researchers therefore call for "provenance-first" auditing of AI-generated research: requiring every numerical claim to link to specific experimental records, metric definitions, and run batches, rather than merely checking surface consistency within the text. For academia and industry rapidly adopting AI research agents, this may be more urgent than simply increasing the sensitivity of "fabrication detection."

After all, a paper citing data from the wrong experiment—even if every number truly exists—may have completely unreliable conclusions.

One-line Summary

Numerical errors in AI agent papers mostly stem from misattribution rather than fabrication, requiring provenance-first auditing.

Paragraph Summary

A re-analysis of 540 AI research agent paper judgments found that numerical errors in these papers are mainly not deliberate fabrication but "misattribution"—citing values from the wrong experiment, wrong metric, or wrong run batch within their own experimental records. About four-fifths of unsupported claims correspond to values existing in the agent's own files, just from the wrong source. Researchers introduced a deterministic "claim-to-evidence" checker with over 90% precision but incomplete coverage; LLM verifiers have higher recall but about three times the false acceptance rate of the deterministic checker, so the two should be combined. Traceability varies significantly across agents, a difference masked by simple pass/fail judgments. Researchers call for a "provenance-first" auditing approach requiring every numerical claim to link to specific experimental records, metric definitions, and run batches.