[KongchangAI]
· 2 min read· 1,380 words

BDBBQ: A New Benchmark for Evaluating Social Bias in Bengali, Revealing LLM Cross-Lingual Blind Spots Through Human-Machine Collaboration

BDBBQ: A New Benchmark for Evaluating Social Bias in Bengali, Revealing LLM Cross-Lingual Blind Spots Through Human-Machine Collaboration

BDBBQ builds a bilingual Bengali bias benchmark, revealing independent dimensions of cross-lingual fairness evaluation.

BDBBQ is a bilingual benchmark for evaluating social bias in Bengali, containing 61,312 English and Bengali QA instances across 12 social categories. Designed with balanced answer ordering, contrastive question framing, and ambiguous/disambiguated contexts, it was validated through 220 statistical tests and four-annotator expert annotation (Fleiss' κ=0.8349). Evaluating eight LLMs across 490,496 instances, the study finds that disambiguated context boosts Bengali accuracy far more than English (+36.59 vs +17.71 percentage points), all models have lower Bengali accuracy, stereotype-aligned bias varies by language and model, accuracy and bias are independent dimensions, and bias is unevenly distributed across categories. The paper argues English-centric fairness evaluation is insufficient for global deployment.

BDBBQ paper title and abstract screenshot

Social bias evaluation of large language models has long been concentrated in the English-speaking world, leaving the cultural contexts of low-resource languages such as Bengali almost entirely in a blind spot. A new paper proposes the BDBBQ benchmark to fill this gap: it transforms Bengali social bias evaluation from scattered exploration into a repeatable, cross-verifiable systematic engineering effort.

Why a Bengali-Specific Bias Benchmark Is Needed

Bengali has a large speaker population, but the sociocultural context of Bangladesh (religion, caste, gender, regional identity, etc.) differs significantly from that of Europe and North America. Directly translating English bias datasets often loses these localized stereotype structures. This is precisely the starting point of BDBBQ: bias is not an abstract linguistic phenomenon but a judgment rooted in specific social soil.

According to the paper, existing bias benchmarks have limited coverage of culturally grounded evaluation for Bengali, especially lacking assessment methods within the sociocultural context of Bangladesh. BDBBQ is therefore designed as a bilingual extension of BanglaBBQ rather than a simple translation.

Dataset Composition: Bilingual, Twelve Categories, Over Sixty Thousand Instances

BDBBQ contains 61,312 question-answer instances, covering both English and Bengali across 12 social categories. The dataset introduces three mechanisms in its design: balanced answer ordering, contrastive question framing, and both ambiguous and disambiguated contexts. Working together, these three mechanisms enable the benchmark to measure both "whether the model knows the answer" and "whether the model is led astray by stereotypes."

Data quality is validated through multiple means: structural checks, 220 statistical tests, and expert annotation of 150 instances by four annotators. Inter-annotator agreement reached Fleiss' κ=0.8349, a strong level of agreement, indicating that the annotation standards are transferable and reproducible.

What Fleiss' κ Means

Fleiss' κ (Fleiss' kappa) is a statistic used to measure the degree of agreement among multiple annotators, applicable to scenarios where three or more annotators classify the same set of samples. Its value typically ranges from 0 to 1: 0 indicates agreement equivalent to random guessing, and 1 indicates perfect agreement. Academia generally regards 0.61–0.80 as "strong agreement" and above 0.81 as "very strong agreement." The κ=0.8349 reported by BDBBQ falls in the very strong agreement range, indicating that the four annotators' criteria for "what constitutes bias" are highly convergent and that the annotation results are not merely individual subjective whims. For a task like bias that inherently carries value judgments, a high κ value is a key prerequisite for the benchmark's credibility—it means that when different research teams reuse this dataset, the bias scores they obtain are comparable.

Five Key Findings from Eight Models

The research team evaluated eight large language models, testing a cumulative 490,496 instances, and arrived at five core conclusions.

Disambiguated Context Helps Bengali More

In disambiguated contexts, the accuracy improvement for Bengali was significantly higher than for English: +36.59 percentage points for Bengali versus +17.71 percentage points for English. This suggests that models rely more on contextual cues to "rescue" themselves in Bengali, while their underlying language understanding is relatively weak.

Ambiguous Context vs. Disambiguated Context

In bias evaluation benchmarks, "ambiguous context" refers to a question where the given information is insufficient to uniquely determine the answer, so the model can only "guess" based on social stereotypes; "disambiguated context" supplements the key information sufficient to pin down the correct answer, giving the question an objectively correct solution. This design draws on the core idea of the BBQ benchmark: by comparing the same template's performance under the two contexts, one can separate two failure modes—"insufficient model capability" and "model led astray by bias." If a model's accuracy rebounds substantially in disambiguated contexts, it possesses the relevant knowledge but defaults to the stereotype shortcut; if the rebound is limited, it is more likely that language understanding itself has shortcomings. The large rebound of +36.59 percentage points for Bengali in BDBBQ precisely indicates that models more frequently rely on stereotypes to answer in ambiguous Bengali scenarios.

All Models Have Lower Accuracy in Bengali

All eight evaluated models generally have lower accuracy in Bengali than in English. The smallest cross-lingual accuracy gap is Mistral-Nemo, at 5.12 percentage points. This result is consistent with the general disadvantage of low-resource languages in LLMs, but specifically for bias evaluation, it means that fairness conclusions on English benchmarks cannot be directly extrapolated.

Stereotype-Aligned Bias Varies by Language and Model

Bias manifestations are not unidirectional. In Bengali, Gemma 2 9B's stereotype-aligned bias increases, while Mistral-Nemo's bias decreases. The same model may exhibit opposite bias patterns in different languages, which makes sweeping conclusions like "model X is fairer" suspect.

Stereotype-Aligned Bias

Stereotype-aligned bias measures the extent to which a model's outputs align with existing social stereotypes. The typical approach is: among multiple candidate answers to the same question, examine whether the model systematically prefers the one consistent with stereotypes (for example, defaulting to associating certain occupations with specific genders or regional groups). This metric is orthogonal to accuracy—a model may exhibit biased tendencies in its answer distribution even when it answers correctly; conversely, an incorrect answer does not necessarily stem from bias. Therefore, BDBBQ reports "stereotype-aligned bias" as an independent dimension rather than deriving it from accuracy. The metric rising and falling differently across languages indicates that bias is not a fixed property of the model but a product of the interaction between the model's internal representations and language- and culture-specific corpora.

High Accuracy Does Not Mean Low Bias

The paper explicitly points out that higher accuracy does not necessarily correspond to lower stereotype-aligned bias. Accuracy and bias are two relatively independent dimensions, and a model's "correct-answer rate" cannot serve as a substitute indicator for fairness.

Bias Is Unevenly Distributed Across Social Categories

The degree of bias varies markedly across social categories. Qwen 2.5 7B, in Bengali, exhibits stronger stereotype alignment in the Politics and Physical Appearance categories. This suggests that evaluation needs to be broken down by category rather than reporting only an overall score.

Significance and Limitations from Multiple Perspectives

The methodological value of this work lies in its "human-machine collaboration" approach to benchmark construction: statistical tests ensure structural validity, expert annotation ensures semantic validity, and the combination of the two is more robust than purely automated generation or purely manual annotation. Some analysts note that BDBBQ's multilingual comparison design provides a reusable template for other low-resource languages; however, others argue that the expert annotation scale of 150 instances is relatively small compared with the overall data volume, and category-level conclusions still require more verification.

The paper's core claim is clear: cross-lingual performance and social bias are two independent dimensions of LLM behavior, and English-centric fairness evaluation is insufficient to support globally deployed models.

Implications for Evaluation Practice

For teams engaged in model evaluation or the deployment of multilingual products, BDBBQ conveys several actionable messages.

First, translation-based benchmarks are insufficient. It is necessary to reconstruct question framing and answer ordering within the target cultural context, as BDBBQ does.

Second, when reporting accuracy, bias metrics must be reported simultaneously, and broken down by language and by social category.

Third, the large improvement brought by disambiguated context is itself a diagnostic signal: if a model relies heavily on contextual disambiguation in a given language, it indicates that the language's foundational representations may not be solid enough.

BDBBQ advances Bengali social bias evaluation to a quantifiable and comparable stage, and also provides a reference engineering paradigm for "culturally grounded fairness evaluation."

One-line Summary

BDBBQ builds a bilingual Bengali bias benchmark, revealing independent dimensions of cross-lingual fairness evaluation.

Paragraph Summary

BDBBQ is a bilingual benchmark for evaluating social bias in Bengali, containing 61,312 English and Bengali question-answer instances across 12 social categories. It is designed with three mechanisms—balanced answer ordering, contrastive question framing, and ambiguous/disambiguated contexts—and validated through 220 statistical tests and four-annotator expert annotation (Fleiss' κ=0.8349). The study evaluates eight large language models across a cumulative 490,496 instances, finding that: disambiguated context improves Bengali accuracy (+36.59 percentage points) far more than English (+17.71); all models generally have lower accuracy in Bengali; stereotype-aligned bias varies in direction by language and model; accuracy and bias are two independent dimensions; and bias is unevenly distributed across social categories. The paper argues that English-centric fairness evaluation is insufficient to support global deployment.

#BDBBQ#Bengali bias benchmark#LLM social bias#cross-lingual fairness#low-resource language evaluation
Share:

Related articles