[KongchangAI]
· 2 min read· 1,195 words

MISO-BR: An Open Corpus for Misogyny Detection in Brazilian Portuguese

MISO-BR: An Open Corpus for Misogyny Detection in Brazilian Portuguese

MISO-BR releases 896 auditable Brazilian Portuguese misogyny annotations; BERTimbau achieves 0.833 macro-F1.

MISO-BR is a misogyny detection resource for Brazilian Portuguese with 896 class-balanced social media texts annotated by three annotators (Fleiss' kappa 0.788). Everything—protocol, guidelines, LLM triage prompts, models, and scripts—is public. Fine-tuned BERTimbau achieves 0.833 macro-F1 vs. 0.651–0.676 for classical models; Base and Large are nearly identical despite a 2.6× training cost gap. Three limitations are disclosed, offering a reproducible template for low-resource sensitive-domain datasets.

Misogynistic speech is one of the most prevalent forms of online hate speech encountered by Brazilian users, yet truly open, auditable resources in Brazilian Portuguese have long been scarce. MISO-BR was released precisely to fill this gap—it is not just an annotated dataset, but a complete, reproducible construction pipeline and evaluation framework.

MISO-BR original release information

What Does an "Auditable" Corpus Mean?

The core claim of MISO-BR is that the value of a resource lies not in impressive leaderboard scores, but in the auditability of its pipeline. This means that every step—from data collection and annotation to model evaluation—has a written protocol and releasable intermediate artifacts.

Specifically, the corpus contains 896 balanced social media texts, all annotated with binary misogyny labels. The release also includes the construction protocol, annotation guidelines, models, and experiment scripts. Three annotators performed manual annotation following written guidelines, achieving Fleiss' kappa of 0.788 on 816 items—a considerable level of agreement for the highly subjective task of hate speech annotation.

Fleiss' kappa is a statistical measure of inter-annotator agreement for scenarios with three or more raters labeling categorical items. Its values typically range from 0 to 1, where 0 indicates agreement equivalent to chance and 1 indicates perfect agreement. Unlike Cohen's kappa, which handles only two annotators, Fleiss' kappa accounts for the possibility that each item is rated by a different set of annotators and corrects for chance agreement. In NLP annotation tasks, 0.61–0.80 is generally considered "substantial agreement" and above 0.81 "almost perfect agreement." Therefore, MISO-BR's 0.788 on 816 items represents a relatively high but not perfect level of agreement, indicating that the annotation guidelines are generally clear, though some room for disagreement remains regarding the subjective boundaries of misogynistic speech.

Where the Data Comes From: Multi-Source Collection and LLM Triage

The corpus construction was not a single-channel crawl. It combined lexicon-guided and target-guided scraping of the X platform with entries imported from OLID-BR. After collection, the data went through a large language model triage stage, whose prompts and decoding parameters are also publicly released.

This combination of "multiple sources + automated pre-screening + manual annotation" balances efficiency and quality control, but it is precisely the aspect the authors emphasize needs scrutiny—because sources influence label distribution, and automated triage may introduce selection bias.

Model Comparison: The Gap Between Classical Classifiers and BERTimbau

Under a unified fixed protocol, the study compared four classical classifiers (Multinomial and Bernoulli Naive Bayes, Linear SVM, Random Forest) with fine-tuned BERTimbau Base and Large.

The results show a clear stratification: classical models cluster around 0.651 to 0.676 macro-F1, biased toward high precision but insufficient recall; Transformer models reach 0.833 macro-F1 and accuracy (with a Wilson 95% confidence interval for accuracy of [0.77, 0.88]), roughly 16 percentage points ahead of classical models.

A notable finding: Base and Large are nearly indistinguishable on the test set, despite a 2.6× difference in training cost. This suggests that in practical deployment, blindly pursuing larger models may not be worthwhile.

BERTimbau is a pretrained BERT model for Brazilian Portuguese, trained by the neuralmind team on large Portuguese corpora such as BrWaC, available in Base (~110M parameters) and Large (~330M parameters) versions. Its key difference from multilingual BERT is that both its vocabulary and pretraining corpus are entirely targeted at Brazilian Portuguese, giving it better coverage of local slang, abbreviations, and social media expressions. In text classification tasks, the standard approach is to attach a classification head on top of BERTimbau and fine-tune all parameters on downstream annotated data. MISO-BR's experiments follow this paradigm and fix the training protocol to ensure comparability with classical classifiers. Note that Base and Large differ in parameter count by roughly threefold, which directly accounts for the 2.6× difference in training cost mentioned in the paper.

Wilson confidence interval is an interval estimation method for binomial proportions (such as accuracy) that performs more robustly than the traditional normal approximation (Wald interval) when the sample size is small or the proportion is near 0 or 1, avoiding out-of-bounds interval endpoints (below 0 or above 1). MISO-BR reports a Wilson 95% confidence interval for accuracy of [0.77, 0.88], meaning that in a repeated-sampling sense, the true accuracy has about a 95% probability of falling within this range. The interval width of approximately 0.11 reflects the estimation uncertainty due to the limited test set size; when comparing Base and Large, if their confidence intervals overlap substantially, one cannot assert superiority based on point estimates alone—which echoes the paper's conclusion that "the two are nearly indistinguishable."

Three Limitations That Must Be Acknowledged

The authors do not avoid the boundaries of the resource but explicitly list three limitations for users to weigh:

Artificial class balance. The corpus deliberately maintains class balance, but the paper provides a quantitative analysis of how this balance affects precision under real-world prevalence rates of misogynistic speech.

Source–label association. There is a correlation between data sources and labels; fortunately, the released source field allows this association to be measured and corrected.

Selection bias from lexicon-guided sampling and automated triage. Both the sampling strategy and LLM pre-screening may cause the corpus to deviate from natural distributions, and users should account for this when interpreting results.

Selection bias in this context means the corpus was not randomly sampled from the natural distribution of the target population but passed through two filters—lexicon-guided sampling and LLM triage—causing the final sample to systematically deviate from real social media text along certain dimensions. For example, lexicon guidance prioritizes posts containing specific keywords, potentially missing misogynistic content that uses subtle expressions or sarcasm; LLM triage may tend to retain "typical" samples while filtering out borderline cases. The consequence is that model performance on the test set may overestimate effectiveness on real traffic. Mitigation strategies include: reporting retention rates at each sampling stage, stratifying evaluation by source, and using the public source field for sensitivity analysis.

Significance for the Brazilian Portuguese NLP Community

Hate speech detection has long been dominated by English resources, and work in low-resource languages often lacks transparency and reproducibility. MISO-BR's contribution lies in providing a decomposable, testable template: protocols public, parameters public, limitations public.

For researchers and engineers, this corpus can serve both as a baseline starting point for misogyny detection and as a reference case for "how to responsibly build datasets in sensitive domains." The corpus, annotation guidelines, models, and experiment scripts are all publicly available.

One-line Summary

MISO-BR releases 896 auditable Brazilian Portuguese misogyny annotations; BERTimbau achieves 0.833 macro-F1.

Paragraph Summary

MISO-BR is a resource for misogyny detection in Brazilian Portuguese, containing 896 class-balanced social media texts annotated by three annotators following written guidelines, with Fleiss' kappa of 0.788. Its core claim is "auditability": the construction protocol, annotation guidelines, LLM triage prompts and decoding parameters, models, and experiment scripts are all public. Data was collected via lexicon-guided scraping, OLID-BR import, and LLM pre-screening before manual annotation. Under a fixed protocol, fine-tuned BERTimbau achieves 0.833 macro-F1 versus 0.651–0.676 for classical classifiers; Base and Large show nearly no difference but differ 2.6× in training cost. The authors explicitly list three limitations—class balance, source–label association, and sampling/triage bias—providing a reproducible template for sensitive-domain datasets in low-resource languages.

#MISO-BR#misogyny detection#Brazilian Portuguese NLP#BERTimbau#hate speech corpus#low-resource languages#reproducible dataset
Share:

Related articles