AI Review Bias: Why LLMs Always Zoom In on Paper Appendices

Study finds GPT and Claude draw about half their review weaknesses from paper appendices, far above the human rate of 8.4%.
A study of 2,000 papers from ACL, EMNLP, COLM, and ICLR compared the focus of LLM and human review comments. It found that about 50% or more of the weaknesses in reviews generated by the latest GPT and Claude models come from paper appendices, versus only 8.4% for humans, whose appendix-derived weaknesses mostly concern presentation and core ideas. AI tends to dig into fine-grained appendix details that are often not central to the paper's contributions. The bias is consistent across models and invocation modes, indicating a systematic feature of LLM reviewing. The study warns that over-reliance on AI weakness lists may divert reviews from core contributions and has released code for reproduction and extension.
In recent years, large language models (LLMs) have gradually permeated the academic peer review process. Many reviewers use AI to identify weaknesses in papers, and its output even directly shapes the final submitted review comments. However, a study of 2,000 papers from ACL, EMNLP, COLM, and ICLR reveals a concerning phenomenon: the latest GPT and Claude models, when generating review comments, draw about 50% or more of their identified weaknesses from paper appendices, whereas the proportion for human reviewers is only 8.4%.

Research Background and Methodology
The study systematically examined the differences in focus between LLM-generated and human review comments. The researchers selected 2,000 papers from four top conferences (ACL, EMNLP, COLM, ICLR) as samples and tested the performance of GPT and Claude models under different usage modes (API calls, coding agents, web access).
The results show that regardless of how the models were invoked, the proportion of weaknesses drawn from appendices in AI review comments remained high. This stands in sharp contrast to human reviewers—only 8.4% of human-identified weaknesses pointed to appendices, and those weaknesses mainly revolved around the paper's presentation and core ideas.
ACL, EMNLP, COLM, and ICLR are representative academic conferences in the fields of natural language processing and machine learning. ACL (Association for Computational Linguistics) and EMNLP (Empirical Methods in Natural Language Processing) are the two oldest and most influential conferences in NLP; COLM (Conference on Language Modeling) is a recently established conference focused on language model research; and ICLR (International Conference on Learning Representations) emphasizes representation learning and deep learning and uses an open review mechanism. The review comments from these conferences are usually publicly available, providing a large-scale, reproducible data foundation for comparing AI and human reviewing differences—an important reason the study chose them as samples.
Differences in Review Focus Between Humans and AI
Human reviewers typically focus more on a paper's core contributions, argumentative logic, and overall presentation. The weaknesses they extract from appendices are mostly related to core content, such as supplementary explanations of key experiments or the clarity of important derivations.
In contrast, LLMs tend to dig deep into appendices for fine-grained technical details, which are often not central to the paper's contributions. This preference can cause AI review comments to overemphasize peripheral issues while neglecting judgments about the paper's overall value.
Why Are AIs "Addicted" to Appendices?
The study does not directly explain the cause of this phenomenon, but it can be inferred from model behavior: appendices usually contain large amounts of technical details, supplementary experiments, and formal proofs, which are higher in text length and complexity and easily attract LLM attention. In addition, models may tend to look for "verifiable" weaknesses, and technical details in appendices fit this pattern better.
One detail: this bias remains consistent across different models and invocation modes, indicating that it is not a coincidence but a systematic feature of LLM reviewing.
From a technical perspective, an LLM's attention mechanism assigns weights to all tokens in the input text. A longer appendix means more technical claims, formulas, and experiment tables, thus providing more surface-level clues for the model to "nitpick." Meanwhile, patterns learned during training prioritize outputting specific, identifiable issues (e.g., an unspecified hyperparameter, a missing baseline), because such statements are easier to verify and articulate than "the paper's core contribution is insufficient." Furthermore, appendices often contain proofs and ablation experiments not elaborated in the main text, which the model may misread as weaknesses the authors have exposed themselves. These factors combine to make the model systematically direct attention toward appendix details in the absence of the ability to judge the paper's overall contribution.
Implications for Academic Peer Review
This finding raises an important warning for the practice of AI-assisted reviewing. If reviewers rely too heavily on weakness lists generated by LLMs, review comments may deviate from the paper's core contributions and become mired in trivial technical details. Over time, this could affect the fairness and effectiveness of academic evaluation.
The researchers hope this analysis will promote academic discussion about AI reviewing and have released the code (https://github.com/cincynlp/review_appendix) for subsequent research to reproduce and extend.
In peer review research, the quality of review comments is usually measured by dimensions such as whether they cover the paper's core contributions, whether the arguments are sufficient, and whether they are constructive. If review comments concentrate heavily on marginal details in appendices, authors may be forced to respond to issues unrelated to the paper's value, and area chairs may find it difficult to judge the paper's true quality. Therefore, when using LLMs as review assistance tools, a more prudent approach is to position them as assistants for initial screening and gap-finding, with human reviewers filtering and reorganizing the weaknesses listed by AI to ensure the final comments return to the paper's core contributions and main argumentative chain.
Conclusion
The efficiency advantages of AI review tools are undeniable, but their bias problems cannot be ignored. Future improvements may include adjusting the model's weighting of appendix content or designing review prompts that focus more on core contributions. In an era where AI is deeply involved in academic evaluation, understanding its limitations is more important than blind trust.
One-line Summary
A study finds that about half of the weaknesses GPT and Claude identify in reviews come from paper appendices, far exceeding the human rate of 8.4%.
Paragraph Summary
A study of 2,000 papers from ACL, EMNLP, COLM, and ICLR compared the focus of LLM and human review comments. Results show that about 50% or more of the weaknesses in reviews generated by the latest GPT and Claude models come from paper appendices, while the proportion for human reviewers is only 8.4%, and the weaknesses humans extract from appendices are mostly related to presentation and core ideas. AI tends to dig into fine-grained technical details in appendices, which are often not central to the paper's contributions. This bias remains consistent across different models and invocation modes (API, coding agents, web access), indicating it is a systematic feature of LLM reviewing rather than a coincidence. The study warns that over-reliance on AI weakness lists may cause reviews to deviate from the paper's core contributions, and it has released code for subsequent reproduction and extension.
Related articles

Codex in Practice: From Installation and Configuration to Multi-Agent Collaboration
Based on a Bilibili Codex course overview, this guide maps the full path of AI coding assistants from installation and agents.md context design to CLI commands, MCP integration, and multi-agent collaboration.

Claude Code Installation and Model Switching Guide for Users in China
How can developers in China install Claude Code reliably? This article compiles a Bilibili creator's hands-on solution: install via NPM, skip official login, use CC Switch to configure API Keys and freely switch between DeepSeek, Kimi, and other models to avoid account bans.

AI Supply Chain Security: What Canary Tokens Reveal
Starting from a Hacker News discussion, this article analyzes the role of canary tokens in AI supply chain security: how they detect abnormal access to model weights, training data, and credentials, as well as coverage blind spots and cost constraints during implementation.