DAR-MGSM
The first benchmark for systematically evaluating the performance of large language models on Arabic-dialect mathematical word problems. It contains two parallel versions, a translated version and a culturally localized version, used to isolate the influence of cultural context on reasoning.
Timeline (last 90 days)
DAR-MGSM is a benchmark for evaluating large language models' performance on Arabic dialect math word problems.
DAR-MGSM contains two parallel versions: a translated version (which only translates the math problems into dialects and MSA) and a translated-and-culturally-localized version (which replaces entities such as names, currencies, and units with regionally common expressions).
Eleven Arabic-centric and multilingual large models were evaluated on DAR-MGSM.
DAR-MGSM provides a reusable evaluation paradigm: when evaluating reasoning ability in low-resource languages, language variant and cultural context should be measured as independent dimensions.
All Facts (4)
DAR-MGSM is a benchmark for evaluating large language models' performance on Arabic dialect math word problems.
50%UnverifiedDAR-MGSM contains two parallel versions: a translated version (which only translates the math problems into dialects and MSA) and a translated-and-culturally-localized version (which replaces entities such as names, currencies, and units with regionally common expressions).
50%UnverifiedEleven Arabic-centric and multilingual large models were evaluated on DAR-MGSM.
50%UnverifiedDAR-MGSM provides a reusable evaluation paradigm: when evaluating reasoning ability in low-resource languages, language variant and cultural context should be measured as independent dimensions.
50%