ICLR Shakes Up Review Scoring: New Four-Tier System Sparks Debate

ICLR Review Scoring Gets Another Overhaul

A reviewer at ICLR recently shared a discovery with the community after picking up three papers to review this cycle: the conference has once again changed its scoring system. Gone are the familiar multi-level granular scores — the new system has been compressed down to just four tiers.

Based on the review interface screenshot shared by the reviewer, the core evaluation question now reads: "Based on your overall assessment of the submission, what is your recommendation? Please take into account the overall soundness, significance, clarity, and contribution of the paper." The available options are:

• 1: Clear rejection
• 2: Weak rejection
• 3: Weak acceptance
• 4: Clear acceptance

Reviewers must now choose between just two broad directions — accept or reject — with only a "strong" or "weak" qualifier for each. The scoring dimension has been dramatically narrowed.

Why Compressing the Score Range Raises Concerns

The reviewer's immediate reaction was that it felt "very strange," arguing that such a drastic compression of the scoring range "doesn't make much sense." This skepticism is not unfounded.

For years, top machine learning venues have used wider scoring ranges (such as 1–10 or multi-tier 1–8 systems) to allow reviewers to express more nuanced stances — whether a paper is a marginal borderline case or a must-accept standout could be clearly signaled through the score. Wider ranges also give Area Chairs (ACs) and Program Chairs richer ordering signals for downstream decision-making.

The new four-tier system eliminates that middle ground and forces reviewers to commit to either "leaning reject" or "leaning accept." Its most significant change is the complete removal of a neutral option: reviewers can no longer assign a "borderline" middle score — they must take a clear stance.

---

Area Chairs (ACs) play a critical role in top-venue review workflows, overseeing a pool of papers and synthesizing feedback from multiple reviewers before recommending accept or reject decisions to Program Chairs. Under a wide scoring system, ACs can quickly identify divergent or clearly strong papers through numerical differences, allowing them to allocate effort efficiently — making quick calls on papers with consistent scores while diving deeper into contentious ones. Program Chairs operate at an even higher level, setting acceptance rates and calibrating global standards. The four-tier system compresses the starting point of this information chain. With ACs and Program Chairs receiving fewer quantitative signals — at a conference where submissions number in the thousands — the impact on process efficiency is hard to ignore.

---

The Likely Intent Behind Four Tiers

Narrowing the score range usually reflects the conference organizers' intent to correct certain patterns in reviewer behavior. Common problems with wider ranges include:

• Score clustering in the middle: Many reviewers default to "safe" middle scores (e.g., 5 or 6), avoiding clear stances and making it harder for ACs to reach decisions.
• Score inflation and standard drift: Different reviewers interpret the same numerical value very differently, undermining cross-reviewer comparability.
• Ambiguous decision signals: When most papers cluster around borderline scores, final acceptance decisions rely more on subjective discussion than on the scores themselves.

The design logic behind mandatory four-tier scoring is precisely to force reviewers to make a decision-oriented judgment after reading a paper: are you actually for or against acceptance? It reframes scoring from "quality rating" to "acceptance recommendation" — a distinction reflected clearly in the wording itself, which asks for a "recommended decision" rather than an abstract quality score.

Potential Impact on Authors and the Review Process

For submitting authors, four-tier scoring means outcomes will feel more binary. Divergence among a paper's three reviewers will be more visibly stark: when scores come in as a combination like 2, 2, 3, the tug-of-war over borderline cases will shift more heavily to the rebuttal stage and the AC's judgment.

At the same time, the loss of fine-grained information may create new problems. Under the old system, a paper scoring 6 versus one scoring 8 were both accept-leaning, but the difference in enthusiasm was immediately visible. Under the new system, both would fall into "weak acceptance" or "clear acceptance," with the distinction left to the written review text rather than the score. This effectively increases reliance on the quality of written reviews — the scores become simpler, but the text must carry far more of the informational weight.

---

The rebuttal stage is a phase in top-venue review workflows that allows authors to submit brief responses after seeing initial reviewer comments, clarifying misunderstandings or adding supplementary experiments. This mechanism exists precisely to handle borderline cases — when reviewer opinions diverge significantly, an author's response can sometimes meaningfully shift a reviewer's final stance. Under four-tier scoring, since the scores themselves no longer convey "how strongly" a reviewer feels, borderline papers with a 2–3 split will rely more heavily on written arguments during the rebuttal phase, placing greater demands on the AC's subjective judgment. This could amplify the impact of genuinely persuasive rebuttals — but it may also magnify the influence of individual AC preferences on final outcomes.

---

Review Mechanisms Continue to Evolve

Notably, this is not the first time ICLR has overhauled its scoring system — the original poster used the word "again" to describe the change, signaling that the conference has been in continuous exploration and iteration on its review mechanisms. As submission volumes grow explosively year after year, top venues are constantly searching for balance among review efficiency, consistency, and fairness — and the design of scoring ranges is a concentrated expression of that ongoing tension.

This change is still in its early stages of community discussion and real-world implementation, and its effects remain to be seen. As the reviewer put it: "but we'll see." Whether the four-tier system ultimately sharpens decision clarity or sparks new controversy over information loss, it reflects the genuine challenges facing academic peer review in an era of massive-scale submissions.

Note: This article is based on a single reviewer's post shared in the Reddit community. For authoritative details, please refer to ICLR's official reviewer guidelines.