AI Bias Audits Can Detect Bias — But Can't Rank Models

A Rigorous Experiment on AI Bias Auditing

As AI regulatory frameworks take shape, bias auditing of high-risk systems is becoming a mandatory requirement. Some regulators and organizations are going further — using audit scores to rank models, reasoning that whichever model shows less bias is more "compliant" and more trustworthy. This logic rests on a hidden assumption: that different audit tools are all measuring the same thing, measuring it accurately enough to be compared against one another.

A new study posted on arXiv (arXiv:2609.15995) directly challenges that assumption. The researchers built a unified inference gateway and ran ten extrinsic audit instruments across a shared panel of ten frontier models, first targeting occupational gender bias and then extending to age and socioeconomic status. The findings are striking: detection succeeded, but ranking failed.

Detection Works — Ranking Is Basically a Coin Flip

On the detection front, the results are robust. Eight of the ten tools successfully detected bias with confidence intervals clearly separated from zero — meaning these tools genuinely "see" bias in the models. The remaining two, widely cited direct-probe benchmarks, suffered from saturation: today's frontier models typically give neutral responses to straightforward questions, rendering these benchmarks nearly useless for detection.

The real problem lies in ranking. The researchers used Kendall's W to measure ranking agreement across tools, and the result was just W=0.07, p=0.83 — statistically indistinguishable from random chance. In other words, if tool A ranks Model X as less biased than Model Y, tool B is just as likely to give you the opposite ordering. The tools can agree on whether bias exists, but they diverge completely on who is more biased.

---

Kendall's W (Kendall's coefficient of concordance) measures ranking agreement across multiple raters, ranging from 0 to 1: W=1 means perfect agreement, W=0 means no agreement — equivalent to random. The W=0.07, p=0.83 result here means that the ten tools' rankings of the ten models are statistically indistinguishable from randomly generated rankings. The p-value far exceeds the conventional significance threshold of 0.05, so we cannot reject the null hypothesis that rankings are entirely random. This isn't just "low agreement" — it's agreement that is statistically indistinguishable from zero, which poses a fundamental challenge to any regulatory or procurement decision that relies on ranked comparisons.

---

Is It Noise — Or Are They Measuring Different Things Entirely?

To rule out the explanation that measurement noise was to blame, the researchers designed a clever positive control: they added six intentionally degraded models with clearly inferior capabilities to the evaluation panel.

This design revealed a critical distinction. When the panel included genuine capability gaps, within-tool reliability recovered — each tool's internal consistency was fine on its own. But even then, cross-tool ranking agreement still did not recover. This rules out the "single construct obscured by noise" explanation, pointing instead to a more fundamental conclusion: these tools are measuring different constructs, not noisy approximations of the same thing.

This is the paper's most important insight. When ten tools that all claim to measure "bias" are each operating with different definitions of bias, aggregating their scores into a single ranking is a conceptual mismatch from the start.

---

Construct is a core concept in psychometrics, referring to an abstract quality or ability being measured — such as "intelligence," "anxiety," or "bias." Construct validity asks: is a measurement instrument actually measuring the construct it claims to measure? When tools that claim to measure the same construct show very low inter-correlations, the standard psychometric interpretation is either excessive measurement error (noise) or that the tools are measuring different constructs. This study's positive control design eliminates the noise explanation — within-tool consistency normalizes once genuine capability gaps are introduced, indicating the tools themselves aren't noisy. The real problem is that each tool operationalizes "bias" differently: some measure word embedding associations, others measure decision-making tendencies, others measure referential patterns in generated text. They capture different facets of bias, not approximate measurements of the same thing.

---

Even the Direction of Bias Reverses Depending on Audit Format

The study also uncovered a counterintuitive finding: the direction of bias itself flips with audit format.

• Forced-choice decision tools predominantly showed "over-correction": in occupational gender tests, models leaned toward female candidates; in hiring decisions, 273 out of 278 cases favored working-class candidates.
• Free generation and default coreference resolution tasks maintained stereotype-congruent tendencies.

The same models, asked in a different way, flip their bias direction from one extreme to the other. This suggests that what we call "bias" is highly dependent on how a tool operationalizes the question, not some stable intrinsic property of the model.

This pattern was replicated on the socioeconomic status dimension. As for the ranking agreement that briefly appeared on the age dimension, it dissolved once the researchers re-examined it against their own tool inclusion criteria.

---

Forced-choice and free generation are two fundamentally different evaluation paradigms. Forced-choice prompts require a model to select from predefined options (e.g., "Choose between candidate A and candidate B"), making the model aware it is choosing and more likely to trigger explicit fairness awareness, leading to over-correction — deliberately favoring the group typically seen as disadvantaged. Free generation lets the model naturally complete a sentence or passage with no obvious "choice" signal, allowing implicit training data biases to surface more readily as stereotype-congruent outputs. The divergence between these two formats indicates that "bias" is largely context-dependent behavior rather than a stable internal parameter, which fundamentally undermines the reasonableness of characterizing a model's bias level with a single audit.

---

What This Means for Regulation and Industry

The practical message from this research is clear — and consequential:

A single audit can detect bias and estimate its direction within its own operationalization, but no single audit can support comparative rankings that place one model above another.

For regulatory approaches that are encoding audit scores into law and using them to rank models, this is a serious warning. Detecting bias and ranking models are two fundamentally different things: the former holds up within a single tool, but the latter lacks any cross-tool basis for comparison. If regulators insist on using one audit framework's scores to rank models and assign compliance status, they are likely amplifying that particular tool's construct preferences — not reflecting genuine differences between models.

On a positive note, the researchers have open-sourced all raw responses, code, and the full analytical pipeline that allows every reported figure to be recomputed from source data on GitHub (williamguey/bias-audit-agreement), providing a strong foundation for reproducibility and follow-up validation.

Conclusion

This paper doesn't dismiss the value of bias auditing — it precisely delineates the limits of what auditing can do: audit tools are effective bias detectors, but unreliable ranking judges. As AI regulation accelerates, that kind of clear-eyed understanding of "what these tools can and cannot do" matters more than a tidy compliance leaderboard. Any attempt to draw conclusions about "who is fairer" from a single audit score needs to revisit its methodological foundations.