ICLR's New Reviewing Policy Sparks Debate: Co-author on 3 Papers? You're a Reviewer Now

The Controversy Over ICLR's Reviewing Policy

A discussion about ICLR's (International Conference on Learning Representations) reviewing policy has been making waves in the Reddit machine learning community. At the heart of the debate is a single rule: if a person's name appears on 3 or more submitted papers, they are required to serve as a reviewer — with no mention of any qualification requirements whatsoever.

The original poster themselves was listed on only 2 papers, so they weren't subject to the policy, but they still questioned its fairness. They offered a concrete hypothetical to illustrate the problem: suppose Alex is a brand-new student who appears as a fourth author on 3 lab papers. Under this rule, would Alex be required to review, even if they're clearly not qualified to do so?

The Logic Behind the Policy — and Where It Falls Apart

This question cuts to a structural problem that top academic conferences have long struggled with: reviewer shortages. As submission volumes in machine learning have exploded, conferences like ICLR and NeurIPS need to mobilize enormous numbers of reviewers every year. Tying reviewing obligations to submission counts is a direct way to ensure adequate reviewer supply — if you're submitting multiple papers to a conference, you should contribute your share of labor to the review process.

From a fairness standpoint, this "submit-then-review" reciprocity makes sense. It prevents the free-rider problem, where a small group of prolific researchers submits paper after paper while offloading the review burden onto others.

However, as the original poster pointed out, the policy falls short on the dimension of qualification. In modern academic collaborations, an author list can span everyone from senior professors to undergraduate interns. A student who contributed to only a small portion of the experiments and appears near the bottom of the author list may lack the academic judgment and domain experience needed to reliably evaluate others' work.

The scale of growth at top ML conferences has been staggering. NeurIPS went from roughly 4,800 submissions in 2018 to over 13,000 in 2023; ICLR grew from around 1,500 to over 7,000 over the same period. With each paper requiring at least 3 reviewers, a single ICLR cycle demands upward of 20,000 person-reviews. The traditional model — relying on voluntary sign-ups and referrals from the Program Committee — has clearly buckled under this growth. The "submission implies reviewing obligation" policy is an institutional response to this reality, and similar rules exist at other top conferences, such as requiring corresponding or senior authors to take on a certain number of reviews.

Ghost Authorship and the Fear of Declining Review Quality

The deeper concern this discussion surfaces is review quality itself. If the policy funnels large numbers of inexperienced junior researchers into the reviewer pool, it risks diluting the overall standard of peer review.

Top ML conferences have already faced mounting criticism over declining review quality in recent years — superficial, perfunctory, or clearly unread reviews are an increasingly common complaint. Mechanically distributing reviewing duties based on paper counts, without regard for actual competence, could make this worse. A first-year student who is forced to review a paper outside their expertise is unlikely to produce a constructive, substantive evaluation.

It's also worth noting that what authorship order means varies widely across labs and cultures. A fourth author in one context might be a core contributor; in another, they may have provided minimal support. Using "name appears on a paper" as the trigger for reviewing obligations is a fairly blunt instrument.

Authorship norms vary significantly across disciplines and cultural contexts — a long-standing controversy in academia. In computer science, the common convention is to order authors by contribution, with the first author typically being the primary executor and the last often being a senior advisor (the "corresponding" or "senior" author), while middle positions remain ambiguous. Bodies like the International Committee of Medical Journal Editors (ICMJE) have proposed "Contributorship" systems that explicitly list each author's specific contributions, distinguishing genuine intellectual contributors from "gift authors" who provided only administrative or resource support. This standard has not been widely adopted in machine learning, making the gap between "name appearing on a paper" and "having the domain judgment to review it" all the more pronounced — and all the more central to this controversy.

What the Conference Organizers May Have in Mind

One important caveat: the original post raised a concern rather than confirmed how the policy is actually enforced. In practice, many conferences still consider factors like a reviewer's publication history and topical fit when assigning specific papers. In other words, "name appearing on 3 submissions" may be a necessary condition for the reviewing obligation, but the conference may well avoid assigning papers clearly beyond a novice reviewer's capabilities.

Furthermore, research teams can typically coordinate internally: a paper usually has multiple authors, and a lab can simply have a more senior member fulfill the reviewing duty rather than leaving a first-year student to handle it alone. The policy language may be broad, but there's operational flexibility in how it's applied.

What This Means for the Academic Community

What looks like a minor procedural debate actually reflects the broader governance challenges the AI research community faces in a period of rapid expansion. Peer review is the gatekeeper of academic quality, and its design requires balancing the need for sufficient reviewers against the need for qualified ones.

For researchers navigating this landscape — especially students new to the field — understanding these policies in concrete terms matters. Knowing what obligations you may incur by being listed as a co-author on multiple papers helps you prepare accordingly. For conference organizers, the next step toward a healthier review ecosystem may lie in more clearly defining eligibility thresholds within the rules, and in offering training and guidance for first-time reviewers.