[KongchangAI]
BenchmarkBanglaBBQ / BDBBQ 基准

BDBBQ

BDBBQ is a bilingual social bias evaluation benchmark covering English and Bengali, containing 61,312 question-answer instances across 12 social categories, designed to evaluate the bias performance of large language models in Bengali sociocultural contexts.

Timeline (last 90 days)

Nov 10

The BDBBQ benchmark contains 61,312 question-answer instances, covering two languages, English and Bengali, and 12 social categories.

Unverified50%
Nov 10

BDBBQ is designed as a bilingual extension of BanglaBBQ rather than a simple translation.

Unverified50%
Nov 10

In the BDBBQ data quality validation, four annotators performed expert annotation on 150 instances, achieving inter-annotator agreement of Fleiss' κ=0.8349.

Unverified50%
Nov 10

In disambiguated contexts, the accuracy improvement is +36.59 percentage points for Bengali and +17.71 percentage points for English.

Unverified50%
Nov 10

The eight evaluated models generally have lower accuracy in Bengali than in English, and among them Mistral-Nemo has the smallest cross-lingual accuracy gap, at 5.12 percentage points.

Unverified50%
Nov 10

In Bengali, Gemma 2 9B's stereotype-aligned bias increases, while Mistral-Nemo's bias decreases.

Unverified50%
Nov 10

The paper explicitly states that higher accuracy does not necessarily correspond to lower stereotype-aligned bias.

Unverified50%
Nov 10

In Bengali, Qwen 2.5 7B exhibits stronger stereotype alignment in the Politics and Physical Appearance categories.

Unverified50%
Nov 10

Stereotype-aligned bias measures the extent to which model outputs align with existing social stereotypes; this metric is orthogonal to accuracy, and BDBBQ reports it as an independent dimension.

Unverified50%
Nov 10

Some hold the view that the scale of expert annotation on 150 instances is relatively small compared with the overall data volume, and category-level conclusions still require more validation.

Unverified50%

1 more timeline events

All Facts (11)

Unverified

The BDBBQ benchmark contains 61,312 question-answer instances, covering two languages, English and Bengali, and 12 social categories.

50%
Unverified

BDBBQ is designed as a bilingual extension of BanglaBBQ rather than a simple translation.

50%
Unverified

In the BDBBQ data quality validation, four annotators performed expert annotation on 150 instances, achieving inter-annotator agreement of Fleiss' κ=0.8349.

50%
Unverified

In disambiguated contexts, the accuracy improvement is +36.59 percentage points for Bengali and +17.71 percentage points for English.

50%
Unverified

The eight evaluated models generally have lower accuracy in Bengali than in English, and among them Mistral-Nemo has the smallest cross-lingual accuracy gap, at 5.12 percentage points.

50%
Unverified

In Bengali, Gemma 2 9B's stereotype-aligned bias increases, while Mistral-Nemo's bias decreases.

50%
Unverified

The paper explicitly states that higher accuracy does not necessarily correspond to lower stereotype-aligned bias.

50%
Unverified

In Bengali, Qwen 2.5 7B exhibits stronger stereotype alignment in the Politics and Physical Appearance categories.

50%
Unverified

Stereotype-aligned bias measures the extent to which model outputs align with existing social stereotypes; this metric is orthogonal to accuracy, and BDBBQ reports it as an independent dimension.

50%
Unverified

Some hold the view that the scale of expert annotation on 150 instances is relatively small compared with the overall data volume, and category-level conclusions still require more validation.

50%
Unverified

The paper's core claim is that cross-lingual performance and social bias are two independent dimensions of LLM behavior, and English-centric fairness evaluation is insufficient to support models deployed globally.

50%

Source Articles