LLMs for Translation Quality Annotation: Who's More Reliable, Humans or Models?

Machine translation (MT) quality evaluation has long relied on human annotation, which is costly and time-consuming. Large language models (LLMs) have been widely expected to replace human judgment and enable more efficient, lower-cost evaluation. However, a new paper systematically tested LLM performance on two mainstream evaluation schemes — Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) — and concluded that across multiple language pairs, domains, and annotation granularities, both humans and models are challenged.

Why Use LLMs for Translation Quality Annotation?

Traditional machine translation evaluation relies on automatic metrics like BLEU and ROUGE, but these metrics have limited correlation with human judgment. MQM and ESA are more fine-grained approaches: MQM requires annotating error types and severity levels, while ESA requires annotating the specific text spans containing errors. Both demand substantial time from professional linguists.

LLMs offer new possibilities — models can quickly read translations, locate errors, and assign severity levels. But the question is: is this capability stable across different language pairs, domains, and annotation granularities? If not, replacing human annotators is out of the question.

The core limitation of automatic metrics like BLEU and ROUGE is that they only perform surface-level matching: BLEU measures n-gram overlap between machine translations and reference translations, while ROUGE is commonly used in summarization tasks and focuses on recall. Such metrics can neither identify semantic errors (e.g., a synonymous substitution being falsely flagged as an error) nor distinguish error severity (omitting a function word and misinterpreting an entire sentence may receive the same penalty). More critically, they require a high-quality reference translation, which is unavailable in many real-world scenarios. MQM and ESA are considered closer to human judgment precisely because they abandon reliance on reference translations and instead directly annotate errors in the translation: MQM classifies and weights errors by type (e.g., accuracy, fluency) and severity (e.g., minor, major), while ESA requires annotators to precisely mark the text spans corresponding to errors. Both are closer to the true meaning of translation quality, but the cost is a sharp increase in annotation effort — which is precisely the motivation for introducing LLMs.

How the Paper Evaluates LLM Annotation Capabilities

The research design covers multiple dimensions:

• Evaluation schemes: MQM (Multidimensional Quality Metrics) and ESA (Error Span Annotation), two mainstream frameworks
• Language pairs and domains: spanning multiple language combinations and text domains
• Annotation granularity: from coarse-grained error types to fine-grained error spans

The core metric is the agreement between LLMs and human annotators. If agreement is high, it suggests models can replace human annotators; if low, it exposes the model's blind spots in specific scenarios. This design is more convincing than testing on a single language pair or domain.

Key Finding: Both Models and Humans Have Weaknesses

The paper's abstract states that LLM performance on these two schemes is neither comprehensively better nor worse than humans — rather, both sides are challenged. This means:

• Models may approach human-level performance on certain language pairs or annotation granularities
• In other scenarios, the agreement between models and humans drops significantly
• Human annotators themselves exhibit inconsistency, which sets a "ceiling" for evaluating models

The comparison between MQM and ESA is also noteworthy. MQM requires understanding error types and severity, while ESA requires precisely locating error spans — the two place different demands on model capabilities. A model may excel at one but perform poorly on the other.

Inter-annotator agreement is a key concept for understanding the paper's conclusions. In human annotation research, multiple annotators are typically asked to independently annotate the same batch of data, and statistical measures such as Cohen's Kappa and Krippendorff's Alpha are used to gauge the degree of agreement among them. If even professional linguists can only achieve moderate agreement with each other, the annotation data itself contains noise, and using it as a "gold standard" to judge models is unfair — a model would be judged as "inconsistent" even if it perfectly replicates human judgment, simply because humans disagree with each other. The paper's emphasis that "both humans and models are challenged" partly reflects this: the prerequisite for evaluating LLMs is first clarifying the reliability boundaries of the human baseline. Moreover, different annotation granularities test different model capabilities — coarse-grained error type classification relies more on semantic understanding, while fine-grained error span localization relies more on boundary judgment and position awareness — and models are likely to differ in their strengths across these two task types.

Implications for Machine Translation Evaluation Practice

This research conveys several practical signals:

Don't blindly replace humans. LLMs can serve as auxiliary tools to quickly screen obvious errors and pre-annotate candidate spans, but final judgment still requires human review, especially in scenarios demanding high precision.

Evaluation must be scenario-specific. LLM reliability varies significantly across language pairs, domains, and annotation schemes. It is dangerous to assume uniformly that "LLMs can replace human evaluation."

Human annotation itself also needs improvement. If agreement among human annotators is insufficient, then using humans as a "gold standard" to judge models has fundamental limitations. The paper's title — "Humans and Models Are Both Challenged" — is precisely a summary of this predicament.

"Model pre-annotation + human review" is a specific form of the human-in-the-loop paradigm, and its feasibility depends on the recall and false positive rates of model pre-annotation. If the model can identify candidate error spans with high recall, human annotators only need to confirm or reject, and overall efficiency can be significantly improved. But if the model misses many subtle errors or frequently generates false positives where there are no errors, the review work may actually take longer than annotating from scratch. In practice, one must also consider the anchoring effect of pre-annotation results on humans — annotators may tend to accept the error types and severity levels suggested by the model, thereby reducing the independence and diversity of annotation. Therefore, when designing hybrid workflows, one must not only evaluate model accuracy but also assess whether it changes human judgment behavior in real workflows and what that change means for final data quality.

Future Directions

For LLMs to be truly usable in machine translation quality annotation, at least three areas of work are needed:

1. Expand test coverage: Validate model performance across more language pairs, domains, and annotation schemes
2. Establish more reliable human baselines: Clarify human annotation agreement and confidence intervals
3. Design hybrid workflows: Model pre-annotation + human review, finding a balance between cost and quality

The current conclusion is cautious: LLMs have potential in translation quality annotation, but they are far from the stage of fully replacing humans. For teams that depend on machine translation quality evaluation, this means designing evaluation workflows more carefully rather than simply handing annotation tasks to models.

One-line Summary

A new paper systematically tests LLM performance on MQM and ESA translation quality annotation, concluding that both humans and models are challenged and full replacement of human annotators is not yet possible.

Paragraph Summary

A new paper systematically evaluates LLM performance on the two mainstream schemes for machine translation quality annotation — Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) — covering multiple language pairs, domains, and annotation granularities. The core conclusion: both LLMs and human annotators are challenged, and models cannot yet fully replace human evaluation. The study finds that LLM-human agreement varies significantly across language pairs, domains, and annotation schemes, requiring scenario-specific reliability assessment; meanwhile, human annotators' own inconsistency sets a ceiling for model evaluation, indicating that the baseline itself needs improvement. In practice, LLMs can serve as pre-annotation auxiliary tools, but high-precision scenarios still require human review — simply replacing humans is a dangerous approach.