Arabic Facebook News Engagement Study: AraEngage Reveals Linguistic Features

Introduction

In social media analysis, engagement prediction features developed for English are often directly applied to languages like Arabic. However, a key characteristic of the Arabic writing system—the omission of short vowels (unvocalized Arabic)—causes many lexical features to shift during measurement. A recent study titled "AraEngage: Interpretable Linguistic Correlates of Engagement in Arabic Facebook News" systematically examines which linguistic indicators correlate with engagement rates in Arabic Facebook news by constructing an interpretable linguistic feature dataset, and deeply audits the fragility of standard dictionary methods in Arabic.

Unvocalized Arabic refers to the phenomenon in Arabic writing where short vowel diacritics (harakat) are typically omitted. Arabic is dominated by consonantal letters; short vowels /a/, /i/, /u/, etc., are only represented by additional marks, but everyday texts (such as news and social media) almost never include these marks. This leads to a large number of homographs: the same string of consonants can correspond to multiple different words, requiring context for disambiguation. For example, كلمة (k-t-b) can be read as "book" (kitāb) or "he wrote" (kataba). For dictionary-based text analysis, this ambiguity directly causes feature misjudgment—a word form may simultaneously match multiple dictionary entries, introducing measurement noise. The AraEngage study is precisely based on this linguistic characteristic to audit the applicability of standard dictionary methods in Arabic.

Dataset and Methods: Construction of AraEngage

The research team created AraEngage, a query-defined sample set containing 10,000 public Arabic Facebook news posts from 2,451 pages across 57 countries, spanning 2023 to 2024. The dataset pairs engagement counts with interpretable linguistic features, enabling researchers to test which Arabic indicators correlate with follower-normalized engagement.

To assess the credibility of these associations, the study considered factors such as publisher context, engagement definitions, and lexical ambiguity. The analysis employed a random forest model (R²=0.38 on post-level splits) and used SHAP for interpretation.

SHAP (SHapley Additive exPlanations) is a model interpretation method based on Shapley values from cooperative game theory, used to quantify each feature's contribution to an individual prediction. In complex models like random forests, SHAP assigns a contribution value to each feature of each sample, with positive values pushing predictions higher and negative values pulling them lower. Its core advantages are local accuracy (the sum of feature contributions equals the difference between the prediction and the baseline) and consistency (feature contributions do not contradict each other due to model changes). In the AraEngage study, SHAP was used to explain the random forest's predictions of follower-normalized engagement, thereby identifying which linguistic features (such as religious expression and message length) are most influential globally or locally, providing a basis for interpretable social media analysis.

Key Findings: Religious Expression, Length, and Lexical Density

In the random forest model, religious expression, message length, sentence length, and lexical density were the linguistic features most correlated with follower-normalized engagement. Notably, unlike studies on English news, positive sentiment was associated with higher engagement in this Arabic news sample.

Message length presents a "length paradox": unadjusted raw engagement was highest in long posts, but the model's attribution for follower-normalized engagement leaned toward short posts. This suggests that simply looking at raw engagement counts may mislead judgments about the role of linguistic features.

Linguistic Audit: Why Standard Dictionary Features Are Fragile

The study conducted a linguistic audit revealing why standard dictionary features are unstable in unvocalized Arabic. For example, the imperative list contains homographs, such as a form readable as "express!" which is also the common preposition "through"; the hedge list conflates the possibility marker qad + imperfective ("might") with the emphatic qad + perfective ("indeed already"); and the religious list mixes pious formulae with place names in political news.

These are plausible sources of measurement error that may weaken the near-zero associations for imperatives and hedges. All findings are observational associations based on operationalized indicators.

Homograph refers to words with the same written form but different meanings or pronunciations. In unvocalized Arabic, homography is particularly prevalent due to the omission of short vowels. For example, the imperative "express!" (عَبِّرْ, ʕabbir) and the preposition "through" (عَبْر, ʕabr) are both written as عبر in unvocalized text, indistinguishable by the consonantal skeleton alone. If a dictionary method lists عبر as an imperative marker, then all uses of عبر as a preposition will also be miscounted as imperatives, causing feature measurement bias. Similarly, qad + imperfective indicates "might" (hedge), while qad + perfective indicates "indeed already" (emphasis); both are written as قد in unvocalized text, and a dictionary that does not distinguish them will conflate hedging with emphatic tone. These homographies and semantic confusions are key reasons why standard dictionary features are fragile in Arabic.

Conclusion and Implications

The AraEngage study emphasizes that language specificity must be considered when applying linguistic features cross-linguistically. For unvocalized languages like Arabic, simple dictionary methods may introduce significant noise. The researchers released derived linguistic features, engagement counts, refined dictionaries, and analysis code (raw posts are not redistributed due to CrowdTangle terms), providing a foundation for subsequent interpretable social media analysis.

The significance of this work lies in reminding researchers and practitioners that the interpretability of engagement prediction models depends on a deep understanding of linguistic details. In engagement analysis of Arabic Facebook news, the roles of religious expression and positive sentiment differ from English contexts, and the effect of message length requires distinguishing between raw and normalized engagement. Future work can further refine dictionaries and explore similar issues in other languages.

Follower-normalized engagement refers to dividing a post's engagement counts (such as likes, comments, shares) by the publishing page's follower count to eliminate the influence of page size differences. In social media analysis, raw engagement counts are highly dependent on an account's follower base; large pages naturally receive more engagement, and direct comparisons conflate the effects of linguistic features with page size. After normalization, the engagement rate is closer to the content's own appeal. The AraEngage study adopted this metric and found a "length paradox" for message length: raw engagement was highest in long posts, but the normalized engagement model leaned toward short posts. This indicates that without normalizing by follower count, the effects of linguistic features may be masked by page size, leading to erroneous conclusions.

One-line Summary

AraEngage built an Arabic Facebook news dataset, revealing the fragility of standard dictionary features in unvocalized Arabic and their engagement associations.

Paragraph Summary

AraEngage constructed a dataset of 10,000 Arabic Facebook news posts from 2,451 pages across 57 countries, spanning 2023-2024. Using a random forest model (R²=0.38) and SHAP interpretation, the study found that religious expression, message length, sentence length, and lexical density were the linguistic features most correlated with follower-normalized engagement. Unlike English studies, positive sentiment was associated with higher engagement in Arabic news. A length paradox emerged: raw engagement was highest in long posts, but the normalized engagement model attributed more to short posts. A linguistic audit revealed that standard dictionary features are fragile in unvocalized Arabic due to homography and semantic confusion—e.g., imperative lists containing prepositional homographs, and hedges conflated with emphatic qad. The study emphasizes that language specificity must be considered when applying linguistic features cross-linguistically, and released derived features, engagement counts, refined dictionaries, and analysis code.