StereoSet
A benchmark dataset for evaluating stereotype bias in language models, covering multiple dimensions including gender, race, occupation, and religion. It measures a model's tendency toward stereotypical content through a multiple-choice question format.
Timeline (last 90 days)
StereoSet通过让模型在刻板印象句、反刻板印象句和无意义句之间做选择，量化模型偏好刻板印象的程度。
Most of the cross-language score drops for Hindi, Korean and Swahili come from specific test items in the StereoSet dataset, rather than from a global fairness improvement.
StereoSet is one of the most commonly used benchmark datasets in multilingual bias auditing, released by Mooney et al. in 2021, specifically designed to measure the degree of stereotyping in language models, with questions in fill-in-the-blank form.
Mainstream bias benchmarks currently include datasets such as WinoBias, StereoSet, and BBQ, which measure a model's stereotyping tendencies along dimensions such as gender and race by constructing contrastive sentence pairs or multiple-choice questions.
A common limitation shared by mainstream bias benchmarks is that they measure a model's explicit choices in specific controlled contexts, rather than implicit biases that emerge in open-ended generation, long-text reasoning, or complex role-playing scenarios.
All Facts (5)
StereoSet通过让模型在刻板印象句、反刻板印象句和无意义句之间做选择，量化模型偏好刻板印象的程度。
50%UnverifiedMost of the cross-language score drops for Hindi, Korean and Swahili come from specific test items in the StereoSet dataset, rather than from a global fairness improvement.
50%UnverifiedStereoSet is one of the most commonly used benchmark datasets in multilingual bias auditing, released by Mooney et al. in 2021, specifically designed to measure the degree of stereotyping in language models, with questions in fill-in-the-blank form.
50%UnverifiedMainstream bias benchmarks currently include datasets such as WinoBias, StereoSet, and BBQ, which measure a model's stereotyping tendencies along dimensions such as gender and race by constructing contrastive sentence pairs or multiple-choice questions.
50%UnverifiedA common limitation shared by mainstream bias benchmarks is that they measure a model's explicit choices in specific controlled contexts, rather than implicit biases that emerge in open-ended generation, long-text reasoning, or complex role-playing scenarios.
50%