Gender Bias in Decision Models: When the Top Answer Decides Everything

Decision models are increasingly used in scenarios such as routing, classification, and triage, returning option probabilities through a single forward pass, while applications act on the top answer. Such models are typically provided as ready-to-use, trained to adapt to the decision format rather than for fairness or specific applications. However, a new study finds that model cards for these models do not include bias measurements, and actual testing reveals concerning gender bias.

Testing Methodology and Subjects

The study tested TypeSafe's closed-source Jev model along with five open-source decision models on gender-swapped yes/no decision tasks, using datasets including Tamkin et al. (2023), BBQ, StereoSet, and the gender bias study by Fulgu and Capraro (2024). These tests aim to assess whether models exhibit gender preferences in decisions involving people.

Log-odds difference is a statistic for measuring the output preference of binary classification models. It takes the difference between the model's log-odds for one option and the log-odds for the other, then compares the shift in that difference under gender swapping. Because log-odds nonlinearly compress probabilities, a difference of +0.05 to +0.10 corresponds to a substantial change in the model's internal relative tendency toward "yes" or "no" answers, not merely jitter in the last digits of probability.

BBQ (Bias Benchmark for QA) is a dataset specifically designed to evaluate social bias in question-answering models, covering dimensions such as gender, race, and age. Questions are divided into ambiguous contexts (insufficient information to determine the answer) and disambiguated contexts. Performance in ambiguous contexts is especially critical: if a model systematically chooses answers consistent with stereotypes on such questions, it indicates reliance on group associations rather than cues in the prompt.

StereoSet quantifies the degree to which a model prefers stereotypes by having it choose among stereotypical, anti-stereotypical, and nonsensical sentences. The "stereotype proportion" mentioned in the text comes from this type of measurement; a value closer to 1 indicates a stronger tendency to choose stereotype-consistent statements.

Core Finding: Women Receive More Favorable Answers

Five of the six models gave significantly more favorable answers to women, with log-odds differences between +0.05 and +0.10. Gender swapping caused 1% to 3.5% of top answers to change, with part of Jev's variation stemming from run-to-run fluctuation. This means that merely changing the gender of the decision subject can alter the model's final output, thereby affecting decision outcomes at the application level.

Side Effects of Confidence Thresholds

Although the confidence thresholds recommended by model manufacturers can remove many biased commitments, on ambiguous BBQ gender questions they increased the proportion of stereotypes among retained answers. Specifically, granite-switch's stereotype proportion rose from 0.76 to 0.85, while pplx's rose from 0.81 to 1.00. This indicates that relying solely on confidence filtering cannot solve the bias problem and may in some cases intensify the concentration of stereotypes.

Confidence threshold refers to the deployment strategy suggested by model manufacturers: adopt the top answer only when its probability exceeds a certain threshold (e.g., 0.7 or 0.8); otherwise hand it off to a human or return uncertain. The intuitive assumption is that "low-confidence outputs are more likely to be wrong, so filtering them out improves reliability." But this mechanism targets answer correctness, not fairness.

When the model itself contains gender associations, high confidence often occurs precisely on samples with strong stereotypes—the model is very "confident" about associations like "nurses are women" and "engineers are men." Therefore, on ambiguous BBQ gender questions, after filtering out low-confidence answers, what remains is precisely the most stereotype-concentrated portion, causing the stereotype proportion among retained answers to rise rather than fall. This shows that confidence and fairness are two orthogonal dimensions and cannot substitute for each other.

Preferences Revealed by Probes

Probe tests further revealed gender preferences in these models: all six models considered violence for the greater good more acceptable against men than against women; two models defaulted to "girl" when forced to guess a child's gender without cues. These preferences reflect gender associations internal to the models.

Probe is a class of methods in model interpretability research: training a simple linear classifier on the activations of a certain layer inside the model to detect whether a specific attribute (such as gender, sentiment, or toxicity) is encoded in that layer's representation. If a linear probe can predict the target attribute from activations with high accuracy, it indicates that the attribute is explicitly represented inside the model.

Unlike directly observing outputs, probes can reveal associations the model "knows but may not say." Findings such as "violence for the greater good is more acceptable against men" and "defaulting children to girls when there are no cues" were extracted from the model's internal preferences through probes or controlled prompting. This suggests that output-level fairness metrics may underestimate gender associations internal to the model, and auditing needs to combine behavioral testing with internal representation analysis.

Auditing Should Match Application Logic

The study finds discrepancies between decision-level and probability-level measurements. Therefore, when auditing models, one should read model outputs the way applications do—focusing on the top answer rather than merely the probability distribution. This is crucial for ensuring the fairness of decision models, especially in high-stakes applications.

This research reminds us that bias in decision models may hide in the top answer, and existing model cards and confidence mechanisms are insufficient to guarantee fairness. Developers and users need more granular auditing methods to identify and mitigate these biases.

One-line Summary

Five of six decision models favor women; confidence thresholds actually exacerbate stereotypes.

Paragraph Summary

The study tested TypeSafe's closed-source Jev model and five open-source decision models on gender-swapped yes/no decision tasks, evaluating bias using datasets including Tamkin et al. (2023), BBQ, StereoSet, and the study by Fulgu and Capraro (2024). Results show that five of the six models gave significantly more favorable answers to women, with log-odds differences of +0.05 to +0.10, and gender swapping caused 1% to 3.5% of top answers to change. Although the confidence thresholds recommended by model manufacturers can filter some biased outputs, on ambiguous BBQ gender questions they actually increased the stereotype proportion among retained answers, e.g., granite-switch rose from 0.76 to 0.85 and pplx from 0.81 to 1.00. Probe tests further showed that all models considered violence against men more acceptable than against women, and some models defaulted children to girls when there were no cues. The study therefore argues that auditing should read the top answer as applications do, rather than looking only at the probability distribution.