Safer for Whom? The Hidden Risks in AI Eating Disorder Interactions

AI assistants offering advice on diet, exercise, and weight loss can potentially reinforce body dissatisfaction, dietary restriction, and compulsive exercise—the core behaviors of eating disorders (ED). A study titled "Safer for Whom? A Systematic Exploration of AI Safety in Eating Disorder Interactions" systematically examines how open-source large language models perform safely when faced with eating disorder-related questions, revealing a problem masked by previous evaluations: models often fail to recognize vulnerability implicit in language, while explicit user information instead becomes the dividing line for safety.

Evaluation Design: Starting from Real Community Questions

Existing assessments tend to focus on diagnostic accuracy, obviously harmful prompts, or synthetic scenarios, making it difficult to answer a more practical question: in natural conversation, can models perceive users' implicit vulnerability and provide consistent protection?

This study takes a different approach. Researchers collected 100 real questions from online pro-anorexia communities, fed them into three open-weight large language models, and scored the responses using clinically informed criteria. The scoring system extends a seven-dimensional taxonomy of eating disorder-related harms, covering not just "saying the wrong thing" but also more subtle harms such as reinforcing restrictive eating and weight stigma.

The key to this design lies in the source of the questions: they are not harmful prompts fabricated by researchers, but question formats circulating in real communities, thus better reflecting models' actual performance in the wild.

Pro-anorexia communities refer to spaces that gather in forms such as online forums and social media groups, whose members view anorexia as a lifestyle or identity rather than a disease, sharing tips and "achievements" related to extreme dietary restriction, purging, excessive exercise, and other behaviors. Such communities often use slang and euphemisms to evade platform moderation, making their question formats highly covert—seemingly ordinary diet or weight-loss inquiries that actually carry eating disorder behavioral patterns. For this reason, real questions collected from such communities can better test models' risk identification capabilities in natural language than manually constructed harmful prompts by researchers, and are closer to the input distribution models may encounter in real usage scenarios.

Safety Performance Varies by Model and Harm Dimension

The results first present a basic fact: safety varies significantly across models and across harm dimensions. No model performs consistently across all dimensions, meaning "whether AI diet advice is safe" cannot be generalized and must be viewed separately by model and by harm type.

More noteworthy is the role of user information. The proportion of high-risk responses is approximately twice as high when no personal bio is provided compared to when one is, indicating that models struggle to identify users' vulnerable states from language alone. When questions are accompanied by explicit eating disorder vulnerability signals, each model's safety improves by 10 to 14 percentage points.

This comparison points to a structural flaw: models' safety mechanisms seem to wait for users to "self-report risk" rather than proactively inferring risk from conversational cues. For users unwilling or unable to explicitly state their condition, this reliance on explicit signals means insufficient protection.

The distinction between explicit vulnerability signals and implicit vulnerability signals is key to understanding this study's conclusions. Explicit signals refer to users directly stating they have or have had an eating disorder, are undergoing treatment, or other clear information; implicit signals are hidden in language style and questioning patterns, such as obsession with calories, pursuit of "clean eating," asking about extreme weight-loss methods under the guise of "a friend," or repeatedly seeking validation for low weight goals. In clinical practice, professionals identify risk precisely through such pragmatic cues. Research shows models mainly rely on explicit signals to trigger safety mechanisms, with insufficient sensitivity to implicit cues—this is related to the fact that large model safety alignment is typically trained on explicitly harmful content. Implicit vulnerability is neither in traditional harm categories nor easily covered by rule-based red-team prompts, thus becoming a blind spot in safety protection.

Safety Disparities Driven by Demographic Characteristics

The study further set matching conditions, adding users' race, gender, and BMI information to questions. Results show that response safety differs significantly across these dimensions, with body size (BMI) causing the largest gap.

Researchers believe these patterns align with a stereotype: eating disorders mainly affect thin white women. When user profiles fall outside this stereotype, model responses may be less protective. In other words, AI's safety net has an invisible "default user," and those deviating from this profile may receive less protection.

This finding advances the AI safety question from "will the model say the wrong thing" to "to whom is the model more likely to say the wrong thing." For topics like diet and weight that are highly tied to identity, safety evaluations that ignore demographic differences may systematically miss the populations most in need of protection.

BMI (Body Mass Index) is a value obtained by dividing weight (kg) by the square of height (m), often used as a rough indicator of weight classification. In the context of eating disorders, BMI has special clinical and social implications: anorexia is often associated with low BMI, but eating disorders also exist widely among normal-weight and high-BMI populations, and diagnoses such as binge eating disorder and atypical anorexia do not presuppose low weight. Including BMI as a user profile variable in evaluation is actually testing whether models presuppose "thin" as a risk prerequisite for eating disorders. If models provide fewer protective responses to high-BMI users, this may reflect a dual bias: both neglecting this group's illness risk and tending to encourage weight loss due to their weight, thereby amplifying harm.

Implications for AI Health Advice Products

From a product perspective, this study suggests several actionable improvement directions. Safety alignment cannot rely solely on users' proactive disclosure; models need the ability to identify eating disorder risk signals from natural language. Meanwhile, evaluation sets should incorporate real community corpora rather than using only synthetic harmful prompts for red-team testing.

For users, the conclusions are also worth noting: the quality of AI advice on diet and weight loss topics doesn't depend on how directly you ask, but on whether the model is "willing" to identify you as someone needing protection. The value of this study lies precisely in bringing this invisible disparity to the surface.

The study is a single preprint work, with conclusions based on three open-source models and 100 community questions. The sample size is limited, and generalization to closed-source commercial models should be cautious.

One-line Summary

Study reveals large models struggle to identify implicit vulnerability signals from eating disorder users, with demographic disparities in safety protection.

Paragraph Summary

This study collected 100 real questions from pro-anorexia communities and used clinically informed criteria to evaluate the safety of three open-source large models in eating disorder-related conversations. Results show that model safety varies by model and harm dimension; the proportion of high-risk responses is approximately twice as high when no user bio is provided, while explicit vulnerability signals only improve safety by 10 to 14 percentage points, indicating models rely on users "self-reporting risk" rather than proactively inferring implicit vulnerability. The study also found significant safety differences across race, gender, and BMI, with body size showing the largest gap, consistent with the stereotype that "eating disorders mainly affect thin white women," suggesting AI's safety net has an invisible "default user."