ICLR Peer Review Meets AI-Generated Papers: How to Spot the Slop?

AI-Generated Papers Are Putting Top Conference Review Systems to the Test

Peer review season for ICLR, one of the premier AI research conferences, is underway — and this year, reviewers are facing a new layer of challenge. A Reddit user sparked community discussion by sharing observations from browsing the ICLR OpenReview platform: while many high-quality submissions exist, a thornier problem has surfaced. AI-generated text is becoming increasingly difficult to detect, and this may be the first top-tier conference to experience "automated research submissions" at scale.

The topic has resonated because it strikes at the very foundation of academic peer review. When large language models can assist — or even drive — the paper-writing process, how can reviewers sift through mountains of submissions to identify genuinely valuable research without being misled by linguistically polished but substantively hollow "slop papers"?

The Core Dilemma Facing Reviewers

The discussion surfaced several issues worth serious reflection. First and foremost is the challenge of detection: advances in AI writing tools have brought machine-generated text ever closer to human writing styles, rendering traditional tell-tale signs like stilted phrasing or logical leaps largely obsolete. The subtle "something's off" instinct that experienced reviewers once relied on can now be smoothed away entirely.

A second issue is the decoupling of writing quality from research quality. A paper that reads fluently and follows proper structure does not necessarily reflect genuine scientific contribution. AI can help authors rapidly generate professional-sounding prose, but it cannot conjure solid experimental data or original ideas out of thin air. This means the focus of peer review must shift even further away from "textual quality" and back toward the scientific core — are the experiments reproducible? Is the methodology truly novel? Do the conclusions hold up?

What Are "Slop Papers"?

The term "slop papers" — referring to low-quality submissions lacking substantive contribution, churned out in bulk or rapidly assembled with AI assistance — recurred throughout the discussion. The danger of such papers lies in the toll they take: they consume enormous reviewer resources, dilute the overall quality of a conference, and make it harder for genuinely excellent research to stand out.

---

ICLR (International Conference on Learning Representations) is one of the most prestigious academic venues in deep learning and representation learning, alongside NeurIPS and ICML as the "big three" AI conferences. ICLR uses the OpenReview platform for open peer review, making all submissions, reviewer comments, and author rebuttals publicly visible. This transparency was designed to improve review quality and deter misconduct — but it also means that review dynamics, including signs of potentially AI-generated submissions, are fully visible to the public. ICLR submission volumes have climbed steadily in recent years, surpassing 7,000 papers in 2024, with 2025 expected to grow further. Given this scale, each reviewer typically handles 6 to 8 papers independently, making the workload already substantial.

---

The Era of Automated Research Submissions Has Arrived

One of the original poster's most forward-looking observations is that ICLR may be the first top-tier conference to encounter "automated research submissions" at scale. "Automated" here refers not just to AI-assisted writing, but to full-pipeline AI involvement — from topic selection and experimental design all the way through to manuscript preparation.

This trend has effects in both directions. On one hand, AI tools genuinely improve research efficiency, helping researchers articulate ideas more quickly and format papers more consistently. On the other hand, as submission barriers drop with widespread tool adoption, submission volumes surge — and the strain on the peer review system grows with them. Top conferences already receive tens of thousands of submissions annually; reviewers are mostly volunteer scholars with limited time and energy, and the burden deepens further when the submission pool may be laced with AI-generated slop.

---

"Automated Research" (also framed as AI-driven Science) is a rapidly evolving concept. In its narrow sense, it refers to using AI tools to assist with individual tasks like literature review, experimental design, data analysis, or manuscript writing. More broadly, it describes end-to-end automation — from hypothesis generation to a publishable paper. Several projects demonstrated early versions of this in 2024, including "AI Scientist"-style systems from institutions like Stanford capable of autonomously selecting topics, writing code, running experiments, and drafting manuscripts. While the output quality of such systems remains inconsistent, their very existence signals something profound: the marginal cost of submitting a paper is approaching zero. The implicit assumption underlying academic publishing — that authors invest significant time and effort — is being quietly undermined.

---

How Peer Review Mechanisms Might Evolve

While the original post raised more questions than it answered, the discussion itself points toward several plausible responses.

Review criteria may place greater emphasis on verifiability. Faced with the near-impossibility of identifying writing origins, conferences may increasingly prioritize hard evidence — open-source code, reproducible experiments, raw datasets — that is difficult to fabricate and serves as a filter against slop.

Review processes may add new checkpoints. For instance, requiring authors during the rebuttal phase to respond to more specific, technically probing questions. Papers generated primarily by AI often falter when pressed for details, making targeted interrogation a natural screening mechanism.

Community norms will need to evolve. How to define the appropriate boundaries of AI use in academic writing, whether to require disclosure of AI involvement, and to what degree — these are questions the entire field must work through collectively.

---

The author rebuttal is a critical stage in top AI conference review pipelines: after reviewer comments are released, authors have roughly one week to respond in writing to the critiques raised, after which reviewers may revise their scores. The mechanism was designed to give authors a chance to correct misunderstandings — but in an era of proliferating AI slop, it may also serve as a "stress test" of genuine research ability. Reviewers who use this stage to ask technically specific questions — ones that require deep domain understanding to answer meaningfully — can often distinguish whether authors truly command their research. Some scholars have already proposed systematically raising the bar for rebuttal-stage questioning: shifting from confirming textual descriptions to probing methodological choices, experimental details, and even failure cases, transforming the rebuttal into an institutional line of defense against AI-generated submissions.

---

Conclusion

This Reddit discussion reflects a very real disruption that the AI wave is delivering to academic peer review. It is not alarmism — it is a challenge already unfolding. As generative AI capabilities continue to advance, "how to distinguish genuine research from AI slop" will become an unavoidable question for every academic conference.

For researchers, rather than worrying about tools being misused, the more productive path is to return to the essence of research itself — truly valuable innovation, rigorous experimentation, and honest scholarly conduct remain core competitive advantages that no AI can replace. For peer review systems, adapting to this shift and rebuilding evaluation standards is no longer a question of "whether to act" but "how to do it better."