What Majority Labels Hide: Annotator Disagreement and Model Errors in Multi-Dialect Arabic Sexism Detection

Research reveals that majority labels obscure annotator disagreement, causing models' error patterns on controversial cases to be overlooked.
The paper builds MuDAS, a multi-dialect Arabic sexism dataset preserving individual annotations, covering YouTube comments in Saudi, Egyptian, and Moroccan dialects. It finds model errors are more frequent on comments with annotator disagreement, and though GPT-5.6 Terra and a fine-tuned encoder have similar Macro-F1 scores, their error patterns are opposite: the former over-classifies, the latter misses implicit and prescriptive sexism. Disagreement-aware learning maintains overall performance and improves consistency on controversial comments, but harms agreed-comment performance and cannot reliably identify controversial cases. Egyptian dialect performs worse, and existing measures cannot fully explain it. The paper calls for treating annotator disagreement as a research signal, not noise.
Introduction: When "Majority Vote" Masks Real Disagreement
In content moderation, sexism detection models typically rely on majority labels in datasets for training and evaluation. However, judgments about sexism are inherently subjective, and significant disagreement can exist between different communities and individuals. A new paper, What Majority Labels Hide: Annotator Disagreement and Model Errors in Multi-Dialect Arabic Sexism Detection, directly addresses this issue: reducing annotators' judgments to a single majority label obscures disagreement and prevents us from seeing how models actually perform on controversial cases.

The study constructs the MuDAS dataset, focusing on YouTube comments in three Arabic dialects—Saudi, Egyptian, and Moroccan. Each comment was annotated by three native speakers, with individual annotation results preserved. Using this dataset, the authors evaluated multiple models, including Arabic encoders fine-tuned with hard labels, soft labels, and perspectivist objectives, GPT-5.6 Terra, and four Arabic-centric large language models.
The MuDAS Dataset: A Multi-Dialect Resource That Preserves Disagreement
The core value of MuDAS lies in "preserving individual annotations." Traditional datasets often retain only majority labels, whereas MuDAS makes the three-annotator results for each comment visible. This provides a foundation for studying the relationship between annotator disagreement and model behavior. Coverage of three dialects (Saudi, Egyptian, Moroccan) also enables cross-dialect comparisons.
The research team used this dataset to answer several key questions: Are model errors correlated with annotator disagreement? How do error patterns differ across models on controversial cases? Can disagreement-aware learning objectives improve model alignment with annotators?
Model Errors Are Highly Correlated with Annotator Disagreement
A core finding is that model errors are more frequent on comments with annotator disagreement. This means that when human annotators disagree, models are also more likely to err. This correlation suggests that majority-label evaluation may underestimate the risk of model failure in the "gray zone."
In content moderation scenarios, controversial comments are precisely the ones that require the most careful handling. If models perform poorly on these cases and evaluation metrics fail to reflect this, post-deployment risks will be systematically overlooked.
Annotator disagreement refers to the phenomenon where multiple annotators assign different labels to the same instance. In subjective tasks (such as sexism judgment), disagreement is not pure noise but may reflect the ambiguity of the case itself or differences in annotators' perspectives. Traditionally, researchers have often compressed disagreement into a single "gold standard" via majority voting, but this discards information about the distribution of disagreement. In recent years, a perspectivist orientation has emerged in natural language processing, advocating for preserving individual annotations and modeling disagreement distributions to evaluate model behavior more granularly. Disagreement-aware learning embodies this approach, attempting to align model outputs with annotator distributions rather than merely fitting majority labels.
Error Patterns of Different Models Are Strikingly Opposite
Although GPT-5.6 Terra and the fine-tuned encoder have relatively close Macro-F1 scores, their error patterns are in sharp contrast.
- GPT-5.6 Terra over-classifies: Even when provided with annotators' definitions of sexism, it still tends to over-label gender-related content as sexism, for example, content offering religious modesty advice.
- The fine-tuned encoder under-detects: Encoder models are more likely to miss implicit and prescriptive sexism.
This contrast shows that a single aggregate score (such as Macro-F1) obscures differences in specific error types across models. For content moderation, over-classification and under-detection have entirely different consequences and require differentiated treatment.
Additionally, the four Arabic-centric large language models overall tend to over-predict sexism and perform worse than the fine-tuned encoder.
Macro-F1 is a commonly used aggregate metric in classification tasks. It first computes precision and recall for each class, then takes the F1 score, and finally averages across all classes. This calculation gives each class equal weight, making it more sensitive to minority classes. However, Macro-F1 is a single scalar and cannot distinguish whether a model "over-classifies" or "under-detects"—the two error patterns may produce similar Macro-F1 values but have vastly different impacts on content moderation systems: over-classification may lead to erroneous removal of normal content, while under-detection may allow harmful content to spread. Therefore, evaluating models solely on Macro-F1 masks structural differences in error types.
Disagreement-Aware Learning: Gains and Losses
The study evaluated three training objectives: hard labels, soft labels, and perspectivist objectives. The results found:
- Disagreement-aware learning can maintain Macro-F1, i.e., overall performance does not degrade.
- Soft labels improve model alignment with annotators on controversial comments, but at the cost of worse performance on comments where annotators agree.
- No paradigm can reliably identify controversial comments.
This means current methods cannot yet teach models to "know what they don't know." In scenarios requiring escalation to human review, this lack of uncertainty identification capability is a key shortcoming.
Hard labels, soft labels, and perspectivist objectives are three different training supervision signals. Hard labels use the single class after majority voting as the target, which is the traditional approach. Soft labels use the distribution of annotator labels (e.g., if two of three annotators label an instance as discriminatory and one as non-discriminatory, the target is [0.67, 0.33]), and use losses such as cross-entropy to make model outputs approximate that distribution. Perspectivist objectives go further, attempting to model each annotator's individual judgment or the structure of disagreement itself, rather than merely fitting an aggregate distribution. The core difference among the three lies in how much they exploit "disagreement": hard labels completely discard disagreement information, soft labels preserve the distribution but assume annotators are exchangeable, and perspectivism attempts to capture systematic differences among annotators.
Egyptian Dialect Performs Worse, Reasons Remain Unclear
The study also found that models perform worse on Egyptian Arabic, a phenomenon most pronounced in large language models. The authors examined test set composition and measures such as dialectness but could not fully explain this gap. This suggests that differences between dialects may involve more complex factors and require further research.
Dialectness is a measure of the extent to which a text exhibits features of a specific dialect. In multi-dialect Arabic scenarios, dialectness may affect model performance: the stronger the dialect features, the greater the divergence from Modern Standard Arabic in training data, and the harder it is for models to handle. However, dialectness is only a coarse indicator; it cannot capture complex differences at the lexical, syntactic, and pragmatic levels, nor can it explain disagreement patterns within annotator groups themselves. The poorer performance on Egyptian dialect may also be intertwined with factors such as data volume, annotator consistency, and topic distribution, and a single measure is unlikely to fully account for it.
Implications for Content Moderation
This study sends a clear signal: majority-label scores may mask contrasting error patterns that are crucial for content moderation. If we rely solely on aggregate metrics, we may miss systematic biases in models on controversial cases.
For teams building multilingual, multi-dialect content moderation systems, preserving individual annotations, analyzing disagreement cases, and distinguishing error types should become routine parts of the evaluation process. Although disagreement-aware learning has potential, current methods still have clear limitations in controversy identification and consistency trade-offs.
The paper's contribution lies in redefining "annotator disagreement" from data noise into a valuable research signal, and demonstrating how it reveals differences in model behavior that are masked by majority labels.
One-line Summary
Research reveals that majority labels mask annotator disagreement, causing models' error patterns on controversial cases to be overlooked.
Paragraph Summary
This paper constructs MuDAS, a multi-dialect Arabic sexism dataset that preserves individual annotations, covering YouTube comments in Saudi, Egyptian, and Moroccan dialects. The study finds that model errors are more frequent on comments with annotator disagreement, and that although GPT-5.6 Terra and a fine-tuned encoder have similar Macro-F1 scores, their error patterns are starkly opposite: the former over-classifies, while the latter misses implicit and prescriptive sexism. Disagreement-aware learning (soft labels, perspectivist objectives) can maintain overall performance and improve consistency on controversial comments, but harms performance on agreed comments and cannot reliably identify controversial cases. Egyptian dialect performs worse, and existing measures cannot fully explain this. The paper calls for treating annotator disagreement as a research signal rather than noise.
Related articles

The 15-Year Programmer Who's Been Unemployed for Two Years: The Tech Worker's Dilemma in the AI Era
A US senior software engineer with 15 years of experience has been unemployed for two years, exhausted his savings, and sought help on Hacker News. This article analyzes the structural causes of long-term unemployment among tech workers, strategies for addressing resume gaps, and the redefinition of experience value in the AI era.

Soundcore Liberty 5 Pro Drops to All-Time Low: The Go-To True Wireless Earbuds for Commutes and Calls
Soundcore Liberty 5 Pro drops to an all-time low of $129.99 (from $169.99). These Anker true wireless earbuds excel at call quality and are on sale at Amazon, Best Buy, and Walmart in four colors—ideal for commutes and online meetings.

After Anthropic's IPO, How Far Can Safety-First Go?
The Verge podcast explores Anthropic's post-IPO core tension: can a public company stay safety-first? How public market pressure reshapes AI lab safety decisions.