Same Question, Different Answers: A User-Persona Bias Audit of Smoking-Cessation Advice Across Six Major LLMs

When millions of people turn to large language model (LLM) chatbots for health advice, a seemingly simple yet hard-to-answer question emerges: does everyone get the same advice? An audit study focused on smoking cessation reveals how model outputs shift systematically depending on "who is asking," by feeding the same questions to six major LLMs while posing as twelve different user personas.

Audit Design: One Set of Questions, Twelve Identities

The study selected six widely deployed LLMs from four providers and tested them in the domain of smoking cessation—an area with clear clinical guidelines and major public health significance. The experiment paired four smoking-cessation-related questions with twelve user personas that varied in life stage, self-described expertise level, and smoking history, ultimately producing 2,880 responses.

The value of this design lies in how it simulates the identity signals present in real-world user-chatbot interactions: when a teenager and a physician ask the same question, does the model deliver medical facts of equal quality? The answer is no.

LLM auditing (algorithm auditing) draws on the audit research tradition in the social sciences: feeding a system controlled queries that differ only in specific attributes and observing whether outputs change systematically, thereby detecting potential unfairness or quality issues. Unlike conventional evaluation (such as benchmarks), auditing does not pursue a single score but focuses on the causal pattern of "input variation—output shift." In this study, the key to the audit design is persona perturbation: keeping the question text completely unchanged and altering only the user persona description, so that any output differences can be attributed to the persona signal rather than the task itself. The strength of this approach is its closeness to real deployment scenarios; its limitation is that persona attributes often vary together, making it difficult to cleanly isolate the independent effect of a single factor.

Key Finding: Adolescents Receive Significantly Less Prescription Medication Information

The most consistent finding of the study is that all six models mentioned FDA-approved prescription smoking-cessation medications less frequently for adolescent personas than for adult personas, with gaps ranging from 32 to 67 percentage points. The direction of this bias was consistent across every model, indicating that it is not an incidental flaw of any single system but a systemic issue across providers.

Another notable pattern is that in five of the six models, personas claiming medical expertise actually received less factual information than personas with no professional background. This runs counter to intuition—people typically expect models to produce richer content for "expert users," but the audit results show that factual coverage does not follow this assumption.

Because personas varied along multiple attributes simultaneously, the researchers cautiously report scenario-level differences rather than attributing effects in isolation to age or expertise alone.

FDA-approved prescription smoking-cessation medications mainly include prescription formulations of nicotine replacement therapy, bupropion, and varenicline. Clinical guidelines typically list varenicline as a first-line medication, but explicitly do not recommend it for adolescents, and safety and efficacy data for some medications in minors are limited. Therefore, the finding that "adolescent personas receive less prescription medication information" requires layered interpretation: it may reflect reasonable conservatism in safety policies during model alignment training, or it may constitute insufficient information—the key question is whether the model appropriately states "this information does not apply to your age group" rather than simply omitting it. This distinction is precisely what the study emphasizes: the difference in mention rates is not itself a conclusion and must be judged alongside clinical appropriateness.

Intervention Test: Can a Single Prompt Close the Gap?

The study further evaluated a deployable intervention: a short system prompt specifying a "factual floor"—the set of core medical facts that every user should receive when relevant—while allowing the model to freely adjust tone, vocabulary, and framing.

The results showed that this prompt narrowed differences between personas in every model, but its effect on overall factual coverage varied in direction and magnitude across models. In other words, a prompt validated on one system cannot be assumed to transfer to another. This is a direct warning for engineering teams hoping to solve fairness issues with a unified prompt.

A system prompt is an instruction injected by the deployer before the user conversation to set the model's behavioral boundaries, role, and output specifications, typically invisible to ordinary users. Unlike prompts in user turns, system prompts have higher instruction priority and are an intervention lever that product teams can centrally control. A "factual floor"-style prompt is a lightweight alignment method: it does not modify model weights but only constrains the minimum set of facts in the output. Its limitation is that different models vary in their degree of instruction-following, safety policies, and knowledge coverage, so the same prompt may have opposite effects on different systems—this is the technical context behind the study's emphasis that "transferability cannot be assumed."

Two Necessary Caveats

The researchers offered two limiting reminders. First, equal mention rates are not always the correct clinical goal: prescription smoking-cessation medications are not recommended for minors, so a defensible "factual floor" must account for age rather than applying a one-size-fits-all standard. Second, because personas varied along multiple dimensions simultaneously, the study reports scenario-level differences rather than isolated effects of age or expertise.

All 2,880 responses and the analysis code have been made public, providing a foundation for subsequent replication and extension of the audit.

Implications for Health AI Applications

The significance of this audit extends beyond the single scenario of smoking cessation. It shows that "equality" in LLM health advice cannot be assumed but requires active auditing; biases introduced by user personas may manifest in both factual density and content selection. For teams building health consultation products, there are at least three actionable directions: establish fact-floor checks stratified by population, validate prompt interventions across models, and distinguish "equal mention rates" from "clinical appropriateness" in evaluation metrics.

One-line Summary

An audit shows that mainstream LLMs systematically provide fewer smoking-cessation facts to adolescents and self-described expert users.

Paragraph Summary

An LLM audit study on smoking cessation fed the same questions to six models from four providers while posing as twelve different user personas, producing 2,880 responses. Results show that all models mentioned FDA-approved prescription smoking-cessation medications 32 to 67 percentage points less often for adolescent personas than for adult personas, with highly consistent direction; five models gave less factual information to personas claiming medical expertise than to those without professional backgrounds. The study also tested a system prompt specifying a "factual floor," finding that it narrowed persona differences but had model-dependent effects on overall factual coverage and could not transfer across systems. The researchers caution that equal mention rates do not equal clinical correctness and that factual floors must account for factors such as age. All responses and code have been made public.