BDBBQ
BDBBQ is a bilingual social bias evaluation benchmark covering English and Bengali, containing 61,312 question-answer instances across 12 social categories, designed to evaluate the bias performance of large language models in Bengali sociocultural contexts.
Timeline (last 90 days)
The BDBBQ benchmark contains 61,312 question-answer instances, covering two languages, English and Bengali, and 12 social categories.
BDBBQ is designed as a bilingual extension of BanglaBBQ rather than a simple translation.
In the BDBBQ data quality validation, four annotators performed expert annotation on 150 instances, achieving inter-annotator agreement of Fleiss' κ=0.8349.
In disambiguated contexts, the accuracy improvement is +36.59 percentage points for Bengali and +17.71 percentage points for English.
The eight evaluated models generally have lower accuracy in Bengali than in English, and among them Mistral-Nemo has the smallest cross-lingual accuracy gap, at 5.12 percentage points.
In Bengali, Gemma 2 9B's stereotype-aligned bias increases, while Mistral-Nemo's bias decreases.
The paper explicitly states that higher accuracy does not necessarily correspond to lower stereotype-aligned bias.
In Bengali, Qwen 2.5 7B exhibits stronger stereotype alignment in the Politics and Physical Appearance categories.
Stereotype-aligned bias measures the extent to which model outputs align with existing social stereotypes; this metric is orthogonal to accuracy, and BDBBQ reports it as an independent dimension.
Some hold the view that the scale of expert annotation on 150 instances is relatively small compared with the overall data volume, and category-level conclusions still require more validation.
1 more timeline events
All Facts (11)
The BDBBQ benchmark contains 61,312 question-answer instances, covering two languages, English and Bengali, and 12 social categories.
50%UnverifiedBDBBQ is designed as a bilingual extension of BanglaBBQ rather than a simple translation.
50%UnverifiedIn the BDBBQ data quality validation, four annotators performed expert annotation on 150 instances, achieving inter-annotator agreement of Fleiss' κ=0.8349.
50%UnverifiedIn disambiguated contexts, the accuracy improvement is +36.59 percentage points for Bengali and +17.71 percentage points for English.
50%UnverifiedThe eight evaluated models generally have lower accuracy in Bengali than in English, and among them Mistral-Nemo has the smallest cross-lingual accuracy gap, at 5.12 percentage points.
50%UnverifiedIn Bengali, Gemma 2 9B's stereotype-aligned bias increases, while Mistral-Nemo's bias decreases.
50%UnverifiedThe paper explicitly states that higher accuracy does not necessarily correspond to lower stereotype-aligned bias.
50%UnverifiedIn Bengali, Qwen 2.5 7B exhibits stronger stereotype alignment in the Politics and Physical Appearance categories.
50%UnverifiedStereotype-aligned bias measures the extent to which model outputs align with existing social stereotypes; this metric is orthogonal to accuracy, and BDBBQ reports it as an independent dimension.
50%UnverifiedSome hold the view that the scale of expert annotation on 150 instances is relatively small compared with the overall data volume, and category-level conclusions still require more validation.
50%UnverifiedThe paper's core claim is that cross-lingual performance and social bias are two independent dimensions of LLM behavior, and English-centric fairness evaluation is insufficient to support models deployed globally.
50%