Is ChatGPT Health Advice Safe? Real Conversations from Four Global South Countries Reveal Risks

When people throw headaches, rashes, medication dosages, or even mental distress at ChatGPT, how safe are the responses they get back? An empirical study of real users in India, Nigeria, Brazil, and Pakistan is the first to evaluate the safety of large language models in health scenarios using real conversation data with age and gender. The conclusion is not encouraging: roughly one in every five health conversations is judged high-risk. Even more alarming, when you feed the user's own historical context into the evaluation model, the high-risk proportion jumps by another 13 to 17 percentage points.

Why Previous Research Missed the Real Risks

Existing safety evaluations of LLM health advice are almost entirely built on clinician-written cases, synthetic prompts, or simulated user personas. These methods are controllable and reproducible, but they have a fatal blind spot: they assume the user is an abstract, decontextualized questioner.

Real-world questioners come with age, gender, prior medical history, and regional constraints on healthcare access. The research team explicitly points out that the Global South has long been absent from this kind of safety evaluation, even though concerns about demographic bias and unsafe advice are well documented in these regions—yet never tested on real usage data.

The study's data comes from a consent-based data donation corpus containing 202,590 conversations from 1,252 users between December 2022 and February 2026, with each conversation linked to the user's self-reported age and gender. Researchers identified 15,555 health conversations from this, used an LLM classifier adapted from established taxonomies to determine health intent, and then scored the safety of responses using two separate standards: "context-blind" and "user-context-aware."

Global South is a commonly used concept in geopolitics and development studies, broadly referring to developing countries and regions in Latin America, Asia, Africa, and Oceania, as opposed to the more industrialized "Global North." The term emphasizes not just geographic location, but systemic differences in historical colonial structures, economic resource distribution, and healthcare infrastructure. In AI health safety research, the Global South is critical because these regions generally face low physician density, poor access to primary care, and widespread informal medication use—users are more likely to treat ChatGPT as a substitute for scarce medical resources rather than a supplement. At the same time, the languages, disease spectra, and medication practices of these regions are often outside the coverage of mainstream model training and evaluation data. The research team's observation that this region has long been absent from safety evaluation essentially points to a structural blind spot: safety standards calibrated on European and American clinical scenarios may not reflect the types of harm experienced by real high-risk populations.

Which Health Issues Carry the Highest Risk

The study provides an intuitive number: roughly one in five health conversations receives a high-risk rating. The risk is not evenly distributed but clearly concentrated in symptom queries and condition consultations.

Researchers identified several recurring failure patterns:

• Unsafe medication or dosage guidance: providing potentially harmful drug recommendations or incorrect dosages
• Harmful self-diagnosis: leading users to diagnose themselves, potentially delaying treatment
• Missing safety warnings: failing to indicate when medical attention is essential
• Encouraging over-reliance on the model: letting users treat ChatGPT as their sole consultation source
• Failure to refer to professionals: not recommending medical attention when it is clearly needed

These failure patterns are not new individually, but when they occur simultaneously under real users, real languages, and real healthcare resource constraints, the cumulative effect becomes dangerous.

Context Awareness Exposes Risks More Thoroughly

The most powerful finding of this study is the difference that switching evaluation perspectives makes.

When evaluators look at only a single conversation (context-blind), a response may appear harmless. But when the user's own accumulated background information is incorporated into the judgment (user-context-aware), the high-risk proportion rises by 13 to 17 percentage points. Even more striking, roughly one-quarter of conversations judged safe in isolated evaluation are actually unsafe for the specific questioner.

What does this mean? A model may give "textbook-correct" generic advice, but for a user with a specific underlying condition, taking a specific medication, or living in an area lacking emergency resources, that advice could directly constitute harm. Synthetic cases can never capture this individualized risk, because synthetic cases don't contain that "specific person."

Researchers draw a direct conclusion from this: safety evaluations built on synthetic cases are very likely to systematically miss these harms.

The difference between context-blind evaluation and user-context-aware evaluation is essentially a difference in the unit of assessment. Context-blind evaluation treats each Q&A exchange as an independent sample, checking only whether the response itself conforms to medical common sense and safety norms; user-context-aware evaluation takes the user's previously accumulated age, gender, prior medical history, medication status, and other historical information as preconditions, then judges whether that response is safe for this specific person. The former is like grading an isolated quiz question; the latter is like a doctor reviewing a complete medical record before making a judgment. The 13 to 17 percentage point gap between the two results shows that safety is not an inherent property of the response text, but a relational property between the response and the recipient's background. This methodological distinction has universal significance for AI evaluation design: any evaluation claiming "safety" must specify at which contextual level the judgment was made.

What This Means for AI Health Applications

This study does not deny the value of ChatGPT for accessing health information, but it advances the question from "is the model right" to "is the model right for this person."

For teams developing health AI products, this study suggests several practical directions: safety evaluation cannot rely solely on clinical case libraries but must incorporate real conversations with demographic backgrounds; referral mechanisms (when to explicitly recommend seeing a doctor) need to be upgraded from soft suggestions to hard rules; and for high-risk intents such as medication, dosage, and self-diagnosis, there should be dedicated protections independent of the general safety layer.

For ordinary users, the data's implications are equally direct: using ChatGPT as a starting point for health information is fine, but treating it as the endpoint—especially when it comes to medication and diagnosis—carries real and quantifiable risks.

Frequently Asked Questions

Which countries does this study cover? Four Global South countries: India, Nigeria, Brazil, and Pakistan, with a total of 1,252 users.

How high is the high-risk proportion? About one in five health conversations is judged high-risk; when user personal background is incorporated, the high-risk proportion rises by another 13 to 17 percentage points.

Why is synthetic case evaluation insufficient? Because roughly one-quarter of responses that appear safe in isolated evaluation are unsafe for the specific questioner. Synthetic cases lack individual background and cannot expose this type of risk.

One-line Summary

Real user data shows about one in five ChatGPT health conversations is high-risk, with risk rising another 13-17 percentage points when personal background is added.

Paragraph Summary

An empirical study covering 1,252 users in India, Nigeria, Brazil, and Pakistan is the first to evaluate LLM safety in health scenarios using real conversation data with age and gender. Researchers identified 15,555 health conversations from 202,590 dialogues. Results show about one in five was judged high-risk, with risk concentrated in symptom queries and condition consultations. Typical failure patterns include unsafe medication dosages, harmful self-diagnosis, missing safety warnings, encouraging over-reliance, and failure to refer to professionals. The most critical finding concerns evaluation perspective: when users' own accumulated background is incorporated, the high-risk proportion rises another 13 to 17 percentage points, and about one-quarter of responses that appear safe in isolated evaluation are actually unsafe for the specific questioner. This means traditional safety evaluations based on synthetic or clinical case libraries systematically miss individualized risks.