EMNLP Submission Guide: A Deep Dive into the ARR Rolling Review Mechanism

Introduction: The Story Behind a Pinned Megathread

In Reddit's NLP academic community, a post titled "EMNLP + ARR Megathread" attracted widespread attention. The moderator made it clear: "Please keep all questions and discussions here — I will remove separately posted threads."

This seemingly simple moderation announcement actually reflects a typical scene during the submission season for top conferences in Natural Language Processing (NLP). Around every EMNLP (Empirical Methods in Natural Language Processing) submission deadline or review result announcement, the community is flooded with repetitive discussions about review progress, score interpretation, and Rebuttal strategies. At the heart of it all is the ARR (ACL Rolling Review) mechanism that has been fully adopted by ACL-affiliated conferences in recent years.

What Are EMNLP and ARR?

EMNLP: A Top-Tier NLP Conference

EMNLP is one of the most influential international academic conferences in natural language processing, often grouped with ACL and NAACL as the "Big Three" in the field. It focuses on empirical methods in NLP, covering topics ranging from machine translation and information extraction to large language models and dialogue systems. For NLP researchers worldwide, publishing at EMNLP is a significant milestone of peer recognition.

EMNLP is organized by SIGDAT (Special Interest Group on Linguistic Data and Corpus-based Approaches to NLP) under ACL (Association for Computational Linguistics). Since its inaugural edition in 1996, it has grown into one of the most important NLP conferences. Unlike the ACL annual meeting, which leans more toward theoretical linguistics foundations, EMNLP emphasizes data-driven and experimentally validated research paradigms — a positioning that has given it enormous influence in the deep learning era. In recent years, EMNLP submissions have surpassed several thousand papers, with acceptance rates typically ranging between 22%-25%, making competition extremely fierce.

As the most authoritative international academic organization in computational linguistics and NLP, ACL oversees multiple top conferences including the ACL Annual Meeting, EMNLP, NAACL (North American chapter), EACL (European chapter), and AACL (Asia-Pacific chapter), as well as publications like the Computational Linguistics journal and TACL (Transactions of the ACL). The launch of ARR was a strategic initiative by ACL to unify the review process across all its conferences, with the goal of building a shared review infrastructure.

ARR: A New Paradigm of Rolling Review

ARR (ACL Rolling Review) is a unified review system introduced by ACL to address numerous pain points in traditional conference reviewing. Before ARR, ACL-affiliated conferences each had independent review processes. This meant that if a paper was rejected from ACL, authors had to wait for the next conference deadline to resubmit, and new reviewers might give entirely different or even contradictory evaluations of the same paper. This "review lottery" phenomenon was widely criticized. Additionally, each conference needed to independently assemble a massive review committee, and top researchers might receive dozens of review invitations per year, leading to reviewer fatigue and declining review quality. Statistics show that ACL 2021 received over 3,000 submissions, making it an increasingly serious challenge to find enough qualified reviewers.

The core philosophy of ARR is to decouple "paper review" from "paper publication":

Rolling reviews: Researchers can submit papers to ARR at any time, and the system assigns reviewers on a monthly basis, rather than being constrained by a single conference deadline.
Reusable reviews: The reviews and meta-reviews a paper receives can be "committed" to a specific conference (such as EMNLP), where area chairs make the final acceptance decision.
Reduced redundant work: When resubmitting after rejection, authors can carry forward previous review records, avoiding evaluation from scratch each time.

The ARR system is built on the OpenReview platform, an open-source academic review management system developed at the University of Massachusetts Amherst. OpenReview was previously widely used by top machine learning conferences like ICLR, known for supporting open review and transparent discussion. In the ARR implementation, OpenReview handles paper submission management, reviewer matching (using semantic similarity algorithms between papers and reviewers), review form customization, and cross-conference review record transfer. This technical choice means ARR inherits both the strengths and limitations of OpenReview — for example, the accuracy of its automatic matching algorithm directly impacts review quality.

This mechanism theoretically improves review consistency and efficiency, but in practice it has also introduced new confusion — which is the fundamental reason for the concentrated burst of community discussions.

Why Are Community Discussions So Intense?

The Complexity of Review Timelines

Under the ARR mechanism, submitters need to track two timelines simultaneously: the ARR review cycle (submission, reviewer assignment, score release, Rebuttal, meta-review) and the target conference's (e.g., EMNLP) Commitment deadline. The interleaving of these two timelines leaves many researchers — especially first-time student submitters — feeling overwhelmed.

Commitment is the critical bridge connecting "review" and "publication" in the ARR system. After a paper completes the ARR review process, authors can choose to "commit" it to a conference that accepts ARR review results. This process is similar to "applying for admission" — the paper carries its complete review record (including reviewer comments, author responses, meta-review summary, and recommendation decision), and senior area chairs or program chairs at the target conference make the final acceptance decision. Notably, conferences have the authority to reject papers that ARR recommends for acceptance, and may also accept papers that ARR did not explicitly recommend, giving conference organizers a degree of discretionary power.

The moderator's decision to establish a centralized Megathread and delete scattered posts was precisely because questions like "When will scores be released?" "How should I write my Rebuttal?" and "What score is needed for acceptance?" surge explosively during specific time periods, seriously disrupting normal information flow in the community.

The Anxiety of Score Interpretation

ARR employs a multi-dimensional scoring system, including Soundness and Excitement among other metrics. This design stems from reflection on the "single overall score" review model. A traditional single score (e.g., 1-10) can obscure a paper's specific strengths and weaknesses, and different reviewers may interpret the same score very differently. ARR's current core scoring dimensions include: Soundness (1-5, measuring methodological correctness and experimental reliability), Excitement (1-5, measuring the importance of the research question and novelty of results), and Overall Assessment. Additional dimensions include Confidence (reviewer self-assessed expertise match) and Reproducibility. Community experience suggests that Excitement scores tend to predict final acceptance better than Soundness, because conferences tend to favor "interesting but flawed" over "correct but mundane" work.

After receiving scores, researchers often fall into collective anxiety about "whether my scores actually have a chance," leading them to seek peer experience in the community. This "crowdsourced" score calibration need is the social-psychological foundation that keeps the Megathread thriving.

The Value and Limitations of Centralized Discussion

Efficiency of Information Aggregation

The greatest benefit of consolidating scattered discussions into a single thread is reduced information retrieval cost. Newcomers can quickly understand the overall situation of the current submission round by scrolling through a single post — whether reviews are delayed, what the score distribution looks like, and what common Rebuttal techniques exist. This self-organized knowledge accumulation, to some extent, compensates for the inadequacy of official communication channels.

Potential Information Bias

However, such community discussions have clear limitations. First is survivorship bias — those willing to share scores publicly tend to be extreme cases with either very high or very low scores. The silence of the middle group distorts the overall perception of the "acceptance threshold." Survivorship Bias is a classic cognitive error in statistics, referring to the tendency to focus only on samples that "survived" a selection process while ignoring the eliminated majority. In academic community discussions, this bias is particularly pronounced: authors with high scores may share out of joy, authors with very low scores may post seeking help out of frustration, while the majority with moderate scores tend to silently observe. This causes the visible score samples in the community to exhibit a bimodal distribution, significantly deviating from the actual score distribution (which typically approximates a normal distribution). Inferring "acceptance thresholds" from these biased samples may overestimate or underestimate the actual bar.

Second is the unofficial nature — the "rules of thumb" circulating in the community are not formal standards from the review committee. Over-reliance on them may lead to misjudgment. Researchers should treat them as reference rather than decision-making criteria.

Practical Advice for Researchers

For researchers currently participating in or about to engage with EMNLP/ARR submissions, several practical lessons can be distilled from this phenomenon:

Plan your timeline in advance: Clearly understand the relationship between the ARR review cycle and the target conference's Commitment deadline, and leave sufficient time for Rebuttal and revisions.
Interpret scores rationally: ARR scores are only a reference. Final acceptance is determined by conference area chairs through comprehensive judgment, and the meta-review often plays a more critical role.
Leverage the community but maintain independent judgment: The Megathread is a great place to get firsthand information, but community sentiment should not be equated with official standards.
Invest in Rebuttal quality: Under the rolling review mechanism, a well-reasoned Rebuttal can significantly change a paper's fate and deserves substantial effort. Rebuttal is the stage in academic conference review where authors are allowed to respond to reviewer comments. Authors typically have 3-7 days to write a brief document clarifying reviewers' misunderstandings, addressing technical concerns, or providing additional experimental results. In the ARR system, Rebuttal occurs after initial reviews are completed but before meta-review, and the author's response is referenced by both reviewers and the Action Editor (equivalent to the meta-reviewer). A high-quality Rebuttal needs to: precisely identify reviewers' core concerns, respond with evidence rather than emotion, and demonstrate the paper's true contributions within limited space. Research suggests that effective Rebuttals can raise scores for approximately 15%-20% of borderline papers.

Conclusion

A brief Reddit moderation announcement reflects the entire NLP academic community's adaptation process amid review mechanism reform. ARR rolling review, as a profoundly significant institutional innovation, is reshaping the way researchers interact with top conferences. The community discussion culture that has organically formed around it embodies both the spirit of mutual help in academia and serves as a reminder: on the road to publishing at top conferences, rational information evaluation and solid research work are always more valuable than anxiety over scores.