Can a 2.5 Overall Score Get Accepted at EMNLP? A Deep Dive into Review Scoring

The Anxiety of a First-Time Submitter

Recently, a researcher shared their first EMNLP submission review results on Reddit and asked the community for advice. This scenario is all too familiar to newcomers in the NLP field—after receiving reviews, faced with a pile of numerical scores, the question everyone wants answered most is always: Does my paper stand a chance of being accepted? Can the Rebuttal phase actually save the day?

The submitter's specific scores were as follows:

Overall: 2.5 / 2.5 / 2.5
Soundness: 3.5 / 3 / 3
Excitement: 3 / 2.5 / 2
Reproducibility: 4 / 3 / 2

Behind this set of scores lie many issues within the current top-tier conference review mechanism that are worth discussing in depth. This article will interpret these scores objectively, drawing on the review standards of the ACL/EMNLP conference series.

Understanding EMNLP's Scoring Dimensions

EMNLP (Empirical Methods in Natural Language Processing) is one of the top conferences in the NLP field, using the same review system as ACL. Only by accurately understanding the meaning of each dimension can you judge a paper's true situation.

It's worth knowing that since its founding in 1996, EMNLP has become one of the three flagship NLP conferences alongside ACL and NAACL, hosted by the SIGDAT special interest group under the ACL (Association for Computational Linguistics). In recent years, as the deep learning wave has driven explosive growth in NLP research, EMNLP's average annual submission volume has exceeded 4,000 papers, creating enormous reviewing pressure. To address this challenge, ACL-affiliated conferences gradually unified and refined their scoring dimensions around 2020, introducing multi-dimensional review metrics such as Soundness, Excitement, and Reproducibility, replacing the previously relatively singular sub-scores, with the goal of making acceptance decisions more transparent and interpretable. Understanding the design logic of this system is a prerequisite for reading the scoring signals.

Overall Is the Core Metric

In the ACL/EMNLP review system, the Overall score is often the key determinant of acceptance. This score typically uses a range from 1 to 5:

Above 4.0: Acceptance is basically assured
3.0–3.5: Borderline, requiring comprehensive judgment by the Area Chair (AC)
2.5 and below: The situation is not optimistic

All three reviewers gave this submitter an Overall of 2.5, sending a fairly consistent "low" signal. The high agreement among the three indicates the paper genuinely has widely recognized weaknesses, rather than a subjective bias of an individual reviewer.

It's worth emphasizing that in the ACL/EMNLP review system, the Overall score is not a simple average of the other dimension scores, but rather an independent, comprehensive judgment by the reviewer of the paper's overall value. This means that even if some dimensions score decently, the Overall may still be low—because when giving the Overall, reviewers factor in aspects not covered by the independent dimensions, such as the paper's potential impact on the field, the completeness of related work, and the quality of writing. Additionally, ACL-affiliated conferences use a Double-Blind Review system, where submitters and reviewers are mutually anonymous. This reduces identity bias to some extent, but it also means the paper must speak through its content alone and cannot rely on the authors' academic reputation to "vouch" for the score.

Soundness: The Methodology Holds Up

Soundness measures the rigor of the paper's methodology, the reasonableness of the experimental design, and whether the conclusions are adequately supported. This author's scores of 3.5/3/3 are decent, indicating that reviewers acknowledge the method itself has no obvious flaws and the experimental logic basically holds. This is a positive signal—methodological problems are often the hardest to remedy during the Rebuttal phase, and here that trouble is absent for now.

In empirical NLP research, Soundness evaluation typically spans several aspects: whether statistical significance testing is adequate (e.g., whether confidence intervals or averaged results across multiple random seeds are reported), whether baseline selection is reasonable and fair, whether ablation studies are complete, and whether the paper's conclusions are supported by data within the claimed scope of applicability. In recent years, with the rise of large language models (LLMs), the choice of evaluation benchmarks has drawn increasing attention—reviewers are increasingly concerned with whether researchers have validated their method's generalization ability across multiple independent datasets, rather than just pushing performance numbers on a single benchmark. This author's decent Soundness performance means their research passed most reviewers' scrutiny on these basic requirements, leaving some room to maneuver in the Rebuttal.

Excitement: The Real Weakness

The problem is concentrated in the Excitement dimension: 3/2.5/2. This dimension considers whether the paper is "eye-catching" and whether it brings a sufficiently novel contribution. One reviewer gave only a 2, which is a fairly negative evaluation.

In top-tier conference reviews, a low Excitement score is often harder to recover from than a low Soundness score. Methodological gaps can be patched with additional experiments and clarifications, but "insufficient novelty" is a rejection of the paper's core value, and a Rebuttal of just a few hundred words is unlikely to reverse the reviewers' overall impression.

The introduction of the Excitement dimension reflects a deeper evolution in top-tier conference review standards. Against the backdrop of explosive growth in NLP research, the ACL Anthology now adds over ten thousand papers annually. When faced with a massive volume of submissions, "is this work worth the community's attention" becomes an increasingly practical filtering criterion. Excitement is essentially an expected-impact assessment—reviewers try to judge whether the paper will be widely cited in the future, whether it opens up a new research direction, or whether it provides a tool or benchmark the community urgently needs. This judgment relies heavily on the reviewer's research vision and subjective preferences, making it one of the most contentious dimensions in the review system. Some researchers criticize that Excitement scoring systematically favors "grand narrative" work while unfairly penalizing solid but understated foundational research. Nonetheless, under the current mechanism, a low score in this dimension remains the most troublesome signal for a submitter.

Reproducibility: Opportunity Behind the Disagreement

The three Reproducibility scores (4/3/2) show significant disagreement. One reviewer strongly approved (4), while another gave only a 2. This kind of disagreement usually means that the disclosure was insufficiently clear regarding experimental details, code availability, hyperparameter descriptions, and so on, leading to markedly different judgments among reviewers. Precisely because of this, it's also the direction with the most actionable room during the Rebuttal phase.

The Reproducibility Crisis has been an important topic in recent years across the entire scientific community, especially in machine learning and NLP. In 2019, a study by Dodge et al. found that a large number of NLP papers lacked sufficient implementation details, making it impossible for independent researchers to reproduce the original results. Since then, ACL-affiliated conferences have begun incorporating Reproducibility as a formal review dimension and encouraging authors to submit a "Reproducibility Package" with complete code and experimental configurations. Specifically, when evaluating this dimension, reviewers focus on whether the hyperparameter search process is transparent, whether the training data preprocessing steps are adequately described, whether sources of randomness (such as random seeds) are controlled, and whether the code is open-sourced or promised to be open-sourced upon acceptance. The extreme divergence between a 4 and a 2 for this author likely indicates ambiguity in the description of some key implementation details—different reviewers interpret the standard for "adequate explanation" differently, or some experimental steps are only mentioned in the appendix and missing from the main text. Such issues can be fully remedied through targeted supplementary explanations in the Rebuttal.

An Objective Assessment of Acceptance Probability

Overall, the acceptance prospects for this set of scores are not optimistic, for two core reasons:

All three reviewers' Overall scores consistently landed at 2.5, with none clearly supporting acceptance (which usually requires 3.5 or above). Without a "champion" reviewer, it's very difficult for the AC to fight for a spot for a borderline paper.
Excitement is weak overall, indicating the paper's contribution failed to impress the reviewers.

Referring to recent ACL/EMNLP acceptance situations, the Overall score threshold for the Main Conference typically needs to approach or exceed 3.0, so a paper with an average Overall of 2.5 has a low probability of getting into the Main Conference. However, there is an important "fallback" here—Findings.

EMNLP has a Findings companion venue, specifically for papers that are methodologically sound but slightly lacking in novelty or impact. Given this author's decent Soundness and primary weakness in Excitement, the chance of making it into Findings is clearly higher than into the Main Conference, and it's worth focusing efforts on.

Findings of ACL (including Findings of EMNLP) was established in 2020, originally motivated by the need to address the increasingly prominent paper backlog problem in the NLP field: a large amount of technically flawless work was rejected due to limited main-conference slots, then re-submitted to other venues for repeated review, wasting enormous academic resources. Findings emerged to provide such papers an officially recognized "soft landing" channel. In terms of academic standing, Findings papers are included in the ACL Anthology, can be cited normally, and can usually participate in exchanges during the main conference in poster or online presentation form—no essential difference from main-conference papers. Some institutions already treat Findings papers as equivalent to main-conference papers when evaluating research output. For researchers submitting to a top conference for the first time, strategically writing the Rebuttal with Findings as the target is a pragmatic and commendable choice—it not only ensures the work gets formally published, but also leaves room for publishing a more complete version later in a journal or another conference.

Can a Rebuttal Really Raise the Score?

This is what submitters care about most. The answer is: Yes, but its effect is limited and requires strategy.

The Real Value of a Rebuttal

The core value of the Rebuttal phase lies not in "persuading reviewers to change their view on novelty," but in the following points:

Clarifying reviewers' misunderstandings: If the low score stems from a misinterpretation of the method, a targeted clarification is most likely to prompt a score increase.
Supplementing experimental data: Provide additional results in response to specific experiments the reviewers questioned.
Resolving Reproducibility disagreements: In this case, Reproducibility showed a clear gap between a 2 and a 4. The author can absolutely try to raise that 2 by adding experimental details, promising to open-source the code, and so on.

The Rebuttal system (Author Response) has about a twenty-year history in top computer science conferences. It was first introduced by machine learning conferences such as NIPS (now NeurIPS), then gradually spread to NLP and AI top conferences like ACL, EMNLP, and ICLR. The core assumption of this mechanism is that low scores based on misunderstandings inevitably arise during review, and giving authors a chance to respond in writing can improve the fairness of acceptance decisions. However, multiple studies on the actual effectiveness of Rebuttals have reached fairly consistent conclusions: on average, a Rebuttal only leads to score adjustments for about 10%–15% of papers, and the magnitude of adjustment is usually very small (within 0.5 points). An analysis by Lourie et al. (2022) of Rebuttal effectiveness at NLP conferences found that successful Rebuttals almost all fall into two categories—"correcting factual misunderstandings" or "providing supplementary experiments explicitly requested by reviewers"—while Rebuttals attempting to persuade reviewers to change their subjective judgment of a paper's importance have an extremely low success rate. This context reminds us that the value of a Rebuttal lies in precisely responding to specific concerns, not in trying to reshape reviewers' overall impression through macro-level argumentation.

Have Clear Expectations

The reality that must be faced is that a low Excitement-type score is extremely hard to reverse through a Rebuttal. Reviewers' judgments about whether a paper is "important or interesting" are subjective and stubborn, and it's very hard to persuade someone who finds the work "not exciting enough" to change their stance with a few hundred words.

Therefore, a rational Rebuttal strategy should be:

Focus on the Reproducibility disagreement, which is the entry point most likely to substantially raise the score;
Provide solid clarifications on Soundness to reinforce the impression that the method is sound;
For Excitement, you can concisely restate the contribution, but don't hold too high expectations.

At the practical level, a high-quality Rebuttal generally needs to satisfy several principles: First, be highly targeted, responding point-by-point to each reviewer's specific questions rather than speaking in generalities; second, be data-backed—for any question that can be answered with experimental results, cited literature, or specific numbers, try to avoid replacing them with purely textual argumentation; third, acknowledge legitimate criticism—candidly admit genuine limitations and explain your revision plan, which wins reviewers' trust better than forced justifications; and finally, control length—ACL/EMNLP usually imposes a strict word limit on Rebuttals (about 500 words), so every sentence must point to a substantive issue and avoid redundant thanks or pleasantries. For the set of scores analyzed here, concentrating the core of the Rebuttal on explaining the Reproducibility disagreement, supplemented by precise responses to Soundness-related concerns, is the most cost-effective strategic choice.

Advice for First-Time Submitters

As a researcher taking your first shot at a top conference, receiving such scores is actually a valuable learning opportunity. The following points are worth considering:

Take every review comment seriously. Even if the paper isn't accepted this time, the reviews are in-depth feedback from domain experts—and free of charge. Analyzing point-by-point "why was novelty scored low" can often help you find a genuine breakthrough for the next version.

View acceptance probability rationally. Top-conference acceptance rates are usually between 20%–25%, so rejection is the norm, not failure. Many later-influential works were also initially rejected. In fact, a widely known phenomenon in the NLP field is that when Attention Is All You Need (the original Transformer paper) was submitted to ICLR 2017, its reviews also contained voices questioning the importance of its contribution, yet the paper ultimately became one of the most-cited papers in the history of deep learning. This reminds us that review results are not equivalent to the final judgment of a work's value.

Make good use of the Findings buffer zone. If the Main Conference is out of reach, being accepted to Findings is still a formal conference publication, well worth fighting for.

Write the Rebuttal seriously but with restraint. Concentrate your energy where scores can genuinely be raised—reproducibility explanations and clearing up misunderstandings—rather than going back and forth with reviewers over subjective taste.

Competition in the NLP field is growing increasingly fierce, and the result of a single submission does not define a researcher's worth. Treating each round of review as an iterative improvement process is the right mindset for long-term growth.