Dialectal Arabic Math Reasoning Benchmark DAR-MGSM Released: How Cultural Localization Challenges Large Models

Mathematical reasoning ability has long been a key yardstick for measuring intelligence in large models, but existing evaluations are almost entirely concentrated on high-resource languages like English, leaving low-resource languages—especially Arabic dialects—nearly unaddressed. Recently, a benchmark called DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) filled this gap. It is the first to systematically evaluate the performance of large language models on dialectal Arabic math word problems, and it reveals an easily overlooked dimension—the challenges posed by cultural localization.

Why Does Math Reasoning in Arabic Dialects Deserve Separate Evaluation?

The Arabic-speaking world exhibits pronounced diglossia: Modern Standard Arabic (MSA) is used for formal writing and education, while everyday communication uses one of five major dialects—Egyptian, Gulf, Levantine, Maghrebi, and Mashriqi. For large models, dialectal text differs enormously from MSA in vocabulary, syntax, and cultural references, and directly applying MSA training data often fails. Math word problems are also more complex than general text—they require multi-step logical reasoning while depending on understanding the entities and scenarios in the problem. If a model is unfamiliar with everyday expressions in a dialect, even strong math ability may stall at the very first step.

The starting point of DAR-MGSM is precisely this: language variation itself significantly affects reasoning performance, and this has been systematically ignored in previous benchmarks.

Diglossia is a core concept in sociolinguistics, referring to a situation where the same language community uses two functionally differentiated language varieties on different occasions. In the Arabic-speaking world, Modern Standard Arabic (MSA) is the vehicle for the Quran, news, education, and formal documents, while everyday speech is carried by regional dialects. The two differ significantly in phonology, vocabulary, and syntax, and can even be regarded as different languages. For language models, MSA corpora dominate the internet overwhelmingly, while dialect data is scarce and spelling is non-standardized, causing distribution shift when models process dialectal input. Math word problems also rely heavily on understanding scenario entities (names, currencies, units), and these expressions in dialects are often completely different from those in MSA, so relying solely on MSA training data makes it hard to generalize to dialect scenarios.

Benchmark Design: Parallel Translated and Localized Versions

The core design of DAR-MGSM is quite ingenious. The researchers constructed two parallel versions:

• Translated-only: Math word problems are literally translated from the source language into five Arabic dialects and MSA.
• Translated & culturally localized: On top of translation, entities such as names, currencies, units of measurement, and everyday scenarios in the problems are replaced with common expressions from the corresponding region.

The problems in the two versions are completely identical in mathematical structure; the only variable is cultural context. This controlled-variable approach allows researchers to cleanly isolate the independent effect of "cultural localization" on model reasoning.

The controlled-variable method in benchmark construction is reflected as follows: the two versions keep the mathematical structure, numerical values, and solution steps completely identical, changing only cultural entities (such as names, currencies, units of measurement, and everyday scenarios). This design allows any performance difference to be attributed to the single variable of cultural localization, rather than problem difficulty or linguistic complexity. In natural language processing evaluation, such parallel controlled designs are often used to separate the effects of linguistic form and semantic content, for example formal vs. semantic evaluation in machine translation. DAR-MGSM transfers this idea to mathematical reasoning, making "cultural context" a quantifiable experimental variable rather than a vague background factor.

Testing Eleven Models: Localized Versions Are Generally Harder

The research team evaluated eleven Arabic-centric and multilingual large models on DAR-MGSM. The overall conclusion is clear: model performance generally declines on culturally localized versions. This means that when problems move from "literal translation" to "cultural adaptation," models need to mobilize additional understanding of regional common knowledge, and reasoning difficulty rises accordingly.

However, the magnitude of the decline varies by model. Models whose training focuses on Arabic show different degrees of sensitivity—some are relatively robust, while others react sharply to localization changes. This suggests that a model's ability to adapt to cultural context does not simply grow with parameter scale, but is closely related to the language distribution and cultural coverage of its training data.

Interpretability Analysis: Similar Accuracy, Divergent Reasoning Paths

More noteworthy is the mechanistic interpretability analysis in the paper. The researchers found that for the same math problem, even when the translated and localized versions yield similar final accuracy, the reasoning trajectories activated inside the model are often completely different. In other words, apparent "correct answers" may mask differences in the underlying reasoning process.

This finding raises a warning for evaluation methodology: looking only at accuracy numbers may underestimate the role of cultural factors. The two versions trigger different computational paths inside the model, showing that cultural context is not irrelevant decoration but a variable deeply involved in constructing reasoning.

Mechanistic interpretability aims to understand a model's computational process by analyzing patterns in internal neurons, attention heads, or activation vectors. Common methods include probing classifiers, causal tracing, and activation patching. In the DAR-MGSM analysis, researchers compared the hidden-state trajectories across layers for the same math problem under the translated and localized versions, and found that even when the final answer was correct, the similarity of intermediate representations could be very low. This indicates that the model may solve superficially identical problems through different "circuits," and accuracy metrics cannot capture such internal differences. This finding echoes a core claim in interpretability research: behavioral equivalence does not equal mechanistic equivalence, and evaluation must combine internal representation analysis to fully assess model capabilities.

Implications for Multilingual AI Evaluation

The significance of DAR-MGSM goes beyond Arabic itself. It provides a reusable evaluation paradigm: when assessing reasoning ability in low-resource languages, language variation and cultural context should be measured as independent dimensions, rather than lumped together under "multilingual performance." For teams advancing products for the Arab market or researching Middle Eastern language understanding, this benchmark also offers a testing tool closer to real usage scenarios—users do not do everyday math problems in MSA; they use the dialects they are familiar with and localized everyday scenarios.

Cross-lingual evaluation of mathematical reasoning is moving from "is the translation accurate enough" to "is the culture grounded enough." DAR-MGSM has brought this issue to the table.

One-line Summary

DAR-MGSM is the first systematic evaluation of large models on dialectal Arabic math word problems, revealing the reasoning challenges posed by cultural localization.

Paragraph Summary

DAR-MGSM is the first math reasoning benchmark covering five major Arabic dialects and Modern Standard Arabic, including both translated and culturally localized versions. Evaluation of eleven models shows that performance generally declines on culturally localized versions, with the magnitude varying by model. Mechanistic interpretability analysis further finds that for the same problem, even when accuracy is similar across the two versions, the model's internal reasoning trajectories often differ significantly. The benchmark suggests that multilingual reasoning evaluation should measure language variation and cultural context as independent dimensions rather than lumping them under multilingual performance.