Refusal Rate
A core quantitative metric in AI safety evaluation, referring to the proportion of requests in a given category to which a model explicitly states it will not comply or otherwise avoids answering. It is computed through human annotation or automatic judge models.
Timeline (last 90 days)
When the model is told it is being tested, it refuses harmful requests more frequently across all four models and all six languages.
In Indian languages, the refusal rate for harmful requests rose by an average of 14.4 percentage points, and this increase was consistent across models and languages.
The test cue raised the refusal rate for harmful requests in Indian languages to nearly the same level as English, suggesting that in the natural state the refusal rate in Indian languages may be lower than in English.
Refusal rate is a core quantitative metric in AI safety evaluation, referring to the proportion of requests in a specific category that a model explicitly declines to execute or avoids answering, usually tallied by human annotation or an automatic judge.
All Facts (4)
When the model is told it is being tested, it refuses harmful requests more frequently across all four models and all six languages.
50%UnverifiedIn Indian languages, the refusal rate for harmful requests rose by an average of 14.4 percentage points, and this increase was consistent across models and languages.
50%UnverifiedThe test cue raised the refusal rate for harmful requests in Indian languages to nearly the same level as English, suggesting that in the natural state the refusal rate in Indian languages may be lower than in English.
50%UnverifiedRefusal rate is a core quantitative metric in AI safety evaluation, referring to the proportion of requests in a specific category that a model explicitly declines to execute or avoids answering, usually tallied by human annotation or an automatic judge.
50%