[KongchangAI]
· 2 min read· 1,393 words

AI Mental Health Recommendation Bias: Audit of 41 Models Reveals Clinician-Patient Recommendation Disparities

AI Mental Health Recommendation Bias: Audit of 41 Models Reveals Clinician-Patient Recommendation Disparities

Audit of 41 language models finds mental health recommendations vary by questioner role and sometimes contradict self-ratings.

An audit of 41 language models simulated mental health questions from clinicians, caregivers, and patients, collecting 35,055 free-text responses plus Likert ratings. Models generally recommended what they rated highly, but gave more detailed advice to clinicians and simplified, hedged advice to lay users. Worryingly, they often recommended interventions they rated below zero, albeit with caveats. The team will release tools and data as a living audit.

Introduction: When Language Models Become Mental Health "Advisors"

A growing number of people are asking language models mental health questions and accepting the answers they provide. But are the recommendations reliable? And does recommendation behavior change depending on who is asking? An audit study of 41 language models sought to answer these questions. The study simulated scenarios in which clinical information reached models from three different footings: a clinician describing a patient, a caregiver describing a loved one, and an individual describing themselves. Using 19 case vignettes and five questions, the study collected 35,055 free-text responses and employed a validated judge to extract every named intervention, while also having 29 models rate the complete option lists for the same cases on a seven-point Likert scale.

Curbside Chatbots study screenshot

Audit Design: Recommendation Differences Across Three Perspectives

The core of the study lies in the manipulation of "footing"—the same clinical information presented in the voice of a doctor, a caregiver, and a self-description. This design captures typical real-world scenarios of user-model interaction: a doctor may seek decision support, a caregiver may seek caregiving advice, and a patient may seek self-management strategies.

Data Scale and Analysis Methods

The study covered 19 case vignettes, with five questions per case, yielding a total of 35,055 free-text responses. Using a validated judge, researchers extracted all named interventions. In addition, 29 models rated the complete option lists for the same cases on a seven-point Likert scale, enabling comparison of whether the models' "recommendation behavior" and "evaluation behavior" were consistent.

Vignette (clinical case scenario) is a standardized tool commonly used in psychology and clinical research, referring to a short, fictional but realistic case description that typically includes key information such as the patient's chief complaint, symptoms, medical history, and social background. In this study, 19 case vignettes were designed to cover a variety of mental health scenarios (such as depression, anxiety, substance use, etc.), and each case was rewritten in the voice of a clinician, a caregiver, and the patient themselves, thereby manipulating the "footing" variable while holding clinical information constant. The advantage of the vignette method is that it is reproducible and comparable, and avoids real patient privacy issues, but it also has the limitation of limited ecological validity—real clinical situations are often more complex and information more fragmented.

The seven-point Likert scale is a commonly used psychometric tool that asks respondents to rate their agreement with a statement or strength of preference between 1 and 7, with the midpoint usually representing neutrality. In this study, 29 models scored each item on the complete list of intervention options, thereby obtaining the models' "evaluation" signal for each intervention. Comparing this structured rating with the "recommendation" behavior in free text is the key to the audit design: the former reflects the model's internal preference ordering, while the latter reflects the advice the model outputs in actual conversation, and the gap between the two reveals the extent to which recommendation behavior may deviate from the model's own judgment.

Key Findings: Recommendations and Evaluations Largely Consistent, but Footing Has a Significant Effect

Overall, models tended to recommend interventions they themselves rated highly. However, recommendation behavior varied by footing: many models provided consultation-style responses to clinicians, while giving hedged, simplified advice to non-expert users. This difference suggests that models may have built-in response strategies for different user roles, but whether these strategies always meet clinical safety requirements warrants further scrutiny.

"Footing" is a concept derived from sociolinguist Erving Goffman's theory of communication, originally referring to the relationship role a speaker projects in discourse relative to their own utterance—for example, "author," "principal," or "animator." In human–language model interaction, footing can be understood as the identity and purpose with which the questioner asks the model: a clinician asks as a professional decision supporter, a caregiver asks as a surrogate help-seeker, and a patient asks as a self-describer. The same clinical information conveys different implicit needs under different footings, and a model with role sensitivity should give responses with different emphases for different footings. The manipulation in this study is based on this theoretical framework, systematically testing whether and how models adjust recommendation strategies according to the questioner's footing.

Consistency and Contradiction Between Recommendations and Ratings

A key finding of the study is that when asked for recommendations for a specific intervention type, models more often listed options they themselves rated below zero as initial choices, usually with caveats attached. This means that in free-text recommendations, models may mention interventions they do not truly endorse, especially when users explicitly ask about a certain type of intervention. Such "conditional recommendations" may lead users to mistakenly believe the intervention is an appropriate initial choice while overlooking the limiting conditions the model attaches.

Differentiated Strategies for Clinicians and Lay Users

Models' responses to clinicians were more consultation-style and may include more detailed clinical reasoning, whereas responses to lay users were more simplified and hedged. This differentiation is not necessarily a flaw—adjusting information complexity for different audiences is reasonable—but there is a need for vigilance against simplified advice potentially omitting key safety warnings.

More concerning is that when answering questions about specific intervention types, models recommended options they themselves rated below zero as initial choices. Although caveats were usually attached, users may focus only on the recommendation itself and ignore the conditions. This suggests that clearer filtering or warning mechanisms are needed in model deployment.

Hedging is a linguistic strategy of using vague qualifiers, in which speakers reduce the certainty of a statement through wording such as "might," "perhaps," or "in some cases" to avoid the risks of absolute assertions. In mental health advice scenarios, hedging is double-edged: on the one hand, moderate hedging aligns with the cautious principles of clinical communication and avoids presenting individualized advice as universal conclusions; on the other hand, excessive or systematic use of hedging with lay users may weaken key safety warnings, making it easy for users to misinterpret conditional recommendations as unconditional advice. This study observed that models used hedged wording more frequently with lay users, a pattern that contrasts with the consultation style of clinician responses, suggesting that models' role-adaptation strategies may inadvertently reduce information clarity for vulnerable users.

Implications for Mental Health AI Applications

This audit study reveals some systematic patterns in language models' mental health recommendations. First, models' recommendation behavior is generally consistent with their evaluation behavior, but contradictions exist on questions about specific intervention types. Second, footing (questioner identity) significantly affects recommendation style and content detail. These findings have reference value for clinical practice, caregiver support, and the design of self-management tools.

The research team plans to release the audit tools, code, and complete corpus upon publication as a "living audit." This open approach helps continuously monitor model behavior and promote more responsible mental health AI applications.

For developers, attention should be paid to the consistency of model recommendations across different user roles, and consideration should be given to adding safety filters when specific intervention types are involved. For users, understanding the limitations of model advice—especially that simplified advice may omit conditions—is crucial.

Conclusion: The Necessity of Continuous Auditing

As language models are increasingly applied in the mental health field, independent auditing becomes a key link in ensuring their safety and effectiveness. The multi-model, multi-footing audit framework provided by this study lays the foundation for subsequent work. The commitment to release tools and corpus will also promote community-wide oversight of model behavior and reduce potential risks.

One-line Summary

An audit of 41 language models found that mental health intervention recommendations vary significantly with questioner footing, and that recommendations are sometimes inconsistent with self-ratings.

Paragraph Summary

An audit study of 41 language models simulated mental health questions posed from three footings—clinician, caregiver, and patient—collecting 35,055 free-text responses and analyzing them alongside seven-point Likert scale ratings. The study found that models generally tended to recommend interventions they rated highly, but recommendation behavior varied by questioner identity: more detailed consultation-style responses for clinicians, and simplified, hedged advice for lay users. More concerning, when asked about specific intervention types, models more often listed options they rated below zero as initial recommendations—though with caveats, users may overlook the limitations. The research team has committed to releasing the audit tools and corpus as a "living audit" for continuous monitoring.

#AI mental health#language model audit#recommendation bias#mental health AI#clinical AI safety
Share:

Related articles