Blind Review Experiment Reveals Over-Reliance on First Scores in Rubrics

Blind Review Experiment Reveals Anchoring Effect in Scoring Rubrics

In model evaluation, inter-rater agreement is often taken as evidence that a rubric is stable. However, a blind review experiment exposed a hidden anchoring effect: when raters could not see previous scores and comments, agreement dropped sharply. Based on a practitioner's share on Reddit, this article explores this phenomenon and its implications for AI evaluation.

Experiment Background: High Agreement Under Non-Blind Review

The team had 3 raters score 480 model outputs, yielding 1,440 ratings in total. In the second and third rounds, previous scores and comments were visible. The report showed that 1,022 of these 1,440 ratings were in agreement. The team therefore concluded that inter-rater reliability was high, the rubric was stable, and human scoring appeared well calibrated. The agreement chart was "very self-congratulatory."

Inter-rater reliability is a statistical measure of how consistently multiple raters score the same item, commonly quantified using Cohen's Kappa, Krippendorff's Alpha, or the intraclass correlation coefficient (ICC). High agreement is typically interpreted as a clear rubric and well-calibrated raters. But agreement metrics only measure whether the results are the same—they do not distinguish whether that sameness stems from a shared understanding of the criteria or from mutual influence among raters. In non-blind settings, later raters can see prior scores and comments, so agreement may be systematically inflated, masking ambiguities in the rubric.

Blind Review Results: A Significant Drop in Agreement

To test the robustness of the rubric, the team ran a blind review on the same dataset. Agreement fell to 749, and 96 rows showed differing interpretations of the rubric. Once the previous comments disappeared, agreement dropped immediately, which shocked the team. Removing score visibility made the raters' anchoring effect impossible to ignore.

Anchoring Effect and the Pitfall of Rubric Calibration

The team realized that their rubric calibration had effectively trained people to align with existing judgments and had hidden where the rubric was ambiguous. For example, in the adjudication queue, there were cases where one rater marked partial compliance as a pass while another marked it as a fail. Such disagreements could have been masked by prior scores in non-blind review.

The anchoring effect was introduced by Tversky and Kahneman in 1974. It refers to people's tendency, when making judgments under uncertainty, to over-rely on the first numerical value or piece of information they encounter as a reference point, even when that reference point is irrelevant to the current judgment. In scoring settings, the previous rater's score becomes the anchor: subsequent raters unconsciously adjust their judgments toward the anchor, and usually by an insufficient amount. Anchoring often co-occurs with conformity and order effects, making it hard to tell whether high agreement under non-blind review reflects "unified standards" or "social influence." Blind review removes the anchor and more faithfully exposes the degree to which raters independently understand the rubric.

How to Distinguish a Stable Rubric from False Agreement?

The team is considering whether to rewrite the rubric, add calibration examples, or replace raters before trusting the next round of agreement data. This question is especially important in AI, where human evaluation is relied upon.

Multiple perspectives: Although this article is based primarily on a single source, the anchoring effect has been extensively studied in psychology, and order and conformity effects in scoring are also common phenomena. In AI evaluation, ensuring rubric clarity and rater independence is a key challenge.

Calibration in evaluation workflows usually refers to bringing multiple raters to a consistent understanding of the rubric through training, examples, and discussion. Calibration itself is an important means of improving evaluation quality, but if raters repeatedly see each other's historical scores during calibration, they may mistake "aligning with existing judgments" for "understanding the rubric." Effective calibration should focus on having raters independently judge boundary cases and explain their reasoning, then compare differences in reasoning—rather than directly aligning scores. The key to distinguishing genuine agreement from false agreement is to compare agreement rates under blind versus non-blind conditions and analyze which scoring dimensions the disagreements cluster in.

Practical Recommendations

1. Conduct blind reviews regularly: In multi-rater scoring, schedule at least one blind round to detect anchoring effects.
2. Clarify the rubric: Reduce ambiguity through calibration examples and training, but be mindful of the conformity pressure calibration can introduce.
3. Analyze disagreement cases: Discuss disagreements from blind review in depth to refine the rubric, rather than simply chasing high agreement rates.
4. Rotate raters: Periodically bring in new raters to avoid groupthink.

Through this experiment, the team learned that high agreement does not necessarily mean a stable rubric—it may simply be the result of anchoring. In AI model evaluation, staying vigilant and using methods such as blind review is essential to obtaining more reliable evaluation results.