Beyond Keyword Search: A New Human-Annotated Benchmark Dataset for Arabic Anxiety Discourse

Keyword search can surface every post containing the word "anxiety," but how many of them are actually expressing personal anxiety? A newly released human-annotated benchmark dataset provides an answer—and exposes a commonly overlooked problem: annotator agreement itself needs to be audited.

The Limits of Keyword Search and Benchmark Construction

In social media research, using keywords to capture relevant discourse is common practice. However, searching Twitter/X with the Arabic keyword "qalaq" (meaning anxiety/worry) returns a mix of personal anxiety expressions, sociopolitical anxiety discussions, and other uses unrelated to anxiety. Keywords alone cannot help researchers distinguish these different contexts.

To address this, the researchers built a benchmark dataset of 1,000 Arabic tweets, all retrieved via the "qalaq" keyword and classified by annotators into six discourse functions. Two native Arabic-speaking annotators independently labeled each tweet, achieving a raw agreement rate of 51.1% and a Cohen's κ of 0.378. This figure is not high in itself, indicating that even native speakers disagree on how to define "anxiety" discourse.

Agreement Auditing: The Overlooked Step in Annotation Quality

The research team did not stop at computing inter-annotator agreement. They further reviewed all 489 disagreement samples and 511 agreement samples, re-examining them against the same annotation guidelines. The result: of the originally agreed-upon labels, 120 (23.5%) were changed after review. In other words, nearly a quarter of the labels that annotators "agreed" on were actually wrong.

This finding challenges conventional practice in annotation quality assessment. Typically, researchers focus only on arbitrating disagreement samples and assume agreement samples are reliable. This study shows that agreement also needs auditing—annotators may share the same misunderstandings or oversights. After review, the final labels formed a consensus reference standard. On a random sample of 200 tweets, another annotator's agreement with this consensus was 60.5% (κ=0.517).

Cohen's κ is a statistical measure of agreement between two annotators, typically ranging from -1 to 1, which corrects for the possibility of chance agreement. It is generally accepted that κ below 0.4 indicates poor agreement, 0.4–0.6 moderate, 0.6–0.8 good, and above 0.8 excellent. In this paper, the raw κ=0.378 falls into the poor range, indicating substantial subjectivity in the annotation task itself—which foreshadows the need to audit agreement samples.

Macro-averaged F1 is a commonly used composite evaluation metric in classification tasks. It first computes precision and recall for each class separately to obtain per-class F1 values, then takes the arithmetic mean across all classes. Unlike micro-averaging, macro-averaging gives equal weight to each class, making it more sensitive to model performance on minority or difficult classes. In this paper, the macro-averaged F1 is 0.784, and the low recall of the residual category drags this value down, highlighting the model's weakness in class-imbalanced scenarios.

Model Performance and Error Attribution

With the consensus reference standard in place, the researchers evaluated large language models on Arabic anxiety discourse classification. The Fanar-C-2-27B model achieved 79.8% accuracy and a macro-averaged F1 of 0.784 (κ=0.749). This looks like a solid result, but error analysis reveals deeper issues.

The model's errors are concentrated in the "residual category," which accounts for 41.1% of all errors, with a precision of 0.905 but a recall of only 0.447. Notably, the research team points out that the definition of the residual category was internally inconsistent in the prompt actually used, so the low recall cannot be entirely blamed on the model itself. This candid attribution is crucial to the fairness of benchmark testing: when class definitions are themselves ambiguous, a model's "errors" may reflect flaws in task design.

Fanar-C-2-27B is an Arabic large language model developed by the Qatar Computing Research Institute (QCRI), part of the Fanar series with 27 billion parameters. The model is specifically optimized for Arabic and performs strongly on Arabic natural language understanding and generation tasks, and is often used as a baseline model in Arabic NLP research. Choosing it as the evaluation target reflects both attention to Arabic-specific models and convenience for comparison with general multilingual models.

The Composition of Anxiety Discourse: Sociopolitical Anxiety Leads

Looking at the annotation distribution, sociopolitical anxiety accounts for 27.1% of the sample, higher than personal emotional expression at 22.0%. This ratio reflects that in the Arabic Twitter/X environment, discussions using "anxiety" as a keyword more often involve sociopolitical issues rather than purely personal emotional venting.

It is important to emphasize that these figures are allocation proportions within the sample, not estimates of overall prevalence or retrieval precision. The researchers explicitly state that these proportions cannot be extrapolated to the true distribution of anxiety discourse on Arabic social media.

Open-Source Contributions and Reproducibility

The study released multiple resources to support future work: annotation guidelines, a residual subtype scheme, fixed data splits, label provenance records, and evaluation code. Making these materials public allows other researchers to compare different models on the same benchmark and facilitates reproduction and auditing of the annotation process.

For researchers interested in Arabic NLP or multilingual discourse analysis, the value of this benchmark lies not only in the data itself but also in its demonstration of a more rigorous annotation quality assurance pipeline—from independent annotation to agreement auditing to the establishment of a consensus reference standard, with a traceable record at every step.

At a time when keyword search still dominates social media research, the core reminder of this work is: retrieved text does not equal target discourse, and annotator agreement does not equal correctness. Both deserve serious attention.

One-line Summary

An Arabic anxiety discourse annotation benchmark reveals the limits of keyword search, with 23.5% of annotator agreement labels changed after review.

Paragraph Summary

This study constructed a benchmark dataset of 1,000 Arabic tweets, all retrieved via the keyword "qalaq" (anxiety) and independently classified by two native annotators into six discourse functions. The raw inter-annotator agreement was 51.1%, with Cohen's κ of 0.378. The team further reviewed all agreement and disagreement samples and found that 23.5% of originally agreed-upon labels were changed after review, showing that agreement also needs auditing. Based on the consensus reference standard, the Fanar-C-2-27B model achieved 79.8% accuracy and 0.784 macro-averaged F1, but errors concentrated in the residual category, whose definition was internally inconsistent in the prompt. Sociopolitical anxiety made up 27.1% of the sample, higher than personal emotional expression at 22.0%. The study released annotation guidelines, data splits, and evaluation code, emphasizing that retrieved text does not equal target discourse and agreement does not equal correctness.