Measuring Risk, Not Harm: A New Framework for Detecting Psychological Harm Risk Factors in LLM Conversations

The psychological harm of large language models is difficult to measure directly—it often only emerges after multiple interactions and is highly dependent on the user's subjective context. A recent study proposes an actionable alternative: measure risk, not harm. Rather than trying to determine whether a user has already suffered psychological harm, the researchers detect observable behaviors in AI responses that constitute psychological harm risk factors, and aggregate these signals to support longitudinal monitoring. This approach provides a scalable measurement layer for AI psychological safety.

Why "Directly Measuring Harm" Doesn't Work

Psychological harm is not like a factual error that can be judged within a single response. It arises with delay and accumulation: the same sentence can affect different users in vastly different ways, and harm may gradually accumulate across repeated interactions rather than suddenly appearing in a single turn. This means any method that relies on single-turn output to judge "whether harm has been caused" inevitably conflates observable AI behaviors with the user's latent, invisible psychological state.

The researchers' core claim is: rather than guessing the user's internal state, take a step back and measure risk factors at the level of AI behavior. These risk factors are objectively observable, annotatable at the response level, and do not depend on model internal states, making them applicable across all models.

Eight Types of Operational Risk Behaviors and a Synthetic Corpus

The researchers operationalized risk factors as eight observable AI behaviors and built a synthetic multi-label corpus accordingly: containing 47,062 AI responses and 373,873 response-factor labels, supplemented by targeted human annotation for validation and evaluation. Multi-label means a single response may simultaneously trigger multiple risk behaviors, which better reflects the complexity of real conversations.

Interestingly, all detectors are model-agnostic, operating only on the LLM's output text without needing access to model internal states. This lowers the deployment barrier and avoids binding to specific model architectures.

Synthetic multi-label corpus refers to an annotated dataset that is not collected directly from real user conversations, but constructed through templates, rules, or generative models; each sample can carry multiple labels simultaneously to characterize the multiple risk behaviors triggered by a single response. This construction approach facilitates controlling label distribution and covering rare combinations, but one must also be wary of the deviation between the synthetic distribution and the real conversation distribution—hence the original paper supplements with targeted human annotation for validation and calibration.

Comparison of Three Detection Approaches and the Cascade Design

The study compared three technical routes: classifiers trained on embeddings, direct LLM judging, and a classifier–judge cascade.

On held-out human-annotated data, the best cascade achieved macro precision 0.86, recall 0.76, and F1 0.79. The value of the cascade design is especially evident in efficiency: on the WildChat dataset, it reduced judge calls by 97.1% relative to direct LLM judging. This means that while maintaining detection quality, the inference cost of large-scale monitoring is substantially compressed.

Another finding comes from DelusionEval: risk factor detection results show clustering within conversations. In other words, risk behaviors are not randomly scattered but concentrated in specific conversations—which precisely supports the longitudinal, conversation-level monitoring approach.

macro precision / recall / F1 are metrics obtained by first computing precision, recall, and F1 separately for each risk factor label, then taking the arithmetic mean across these labels. Unlike micro averaging (computed after aggregating by sample or label instance), macro averaging treats each label equally, so rare risk factors and high-frequency risk factors carry the same weight. This makes macro F1 better reflect the detector's balanced performance on long-tail risk behaviors, rather than being dominated by high-frequency labels.

Classifier–judge cascade is a cost-tiered architecture: first, a lightweight, inexpensive classifier screens all responses, and only samples where the classifier's confidence falls in a gray zone or is judged high-risk are sent to the expensive LLM judge for review. The 97.1% call reduction in the original paper comes precisely from this routing—the vast majority of low-risk responses are cleared at the screening stage without triggering large-model inference.

Implications for AI Safety Monitoring

The contribution of this work is not to provide a "psychological harm determiner," but to establish a layered measurement logic: response-level risk behavior detection → conversation-level signal aggregation → longitudinal psychological safety monitoring. It clearly distinguishes between "observable AI behaviors" and "latent user harm," avoiding the common fallacy of equating the former directly with the latter.

For teams deploying conversational AI products, this framework offers a pragmatic engineering path: without invading user privacy or speculating about user psychological states, a risk signal layer can be built solely from output text, and limited high-cost judging resources can be concentrated on high-risk conversations. The researchers also note that final confirmation of harm still requires user context; risk detection is only the starting point of the monitoring chain, not the endpoint.

WildChat is a public dataset of real user interactions with ChatGPT, containing multilingual, multi-scenario natural conversations, often used as an external validation set for detectors on real distributions. DelusionEval is an evaluation set focused on delusion-related content, used to examine whether risk signals cluster in specific conversations. The two serve different purposes: the former measures the method's generalization and efficiency on open real-world traffic, while the latter is used to observe the temporal and clustering structure of risk factors within conversations.

One-line Summary

Research proposes measuring observable risk behaviors in AI responses rather than directly judging psychological harm, enabling scalable longitudinal monitoring.

Paragraph Summary

Psychological harm is delayed and cumulative, making it difficult to judge directly within a single response. This study advocates stepping back to measure risk factors at the level of AI behavior, operationalizing risk as eight observable AI behaviors and building a synthetic multi-label corpus with 47,062 responses and 373,873 labels. The study compares three routes—embedding classifiers, direct LLM judging, and a classifier–judge cascade—with the best cascade achieving macro precision 0.86, recall 0.76, and F1 0.79 on human-annotated data, and reducing judge calls by 97.1% on WildChat. Risk factors cluster within conversations, supporting conversation-level longitudinal monitoring. The framework distinguishes observable AI behaviors from latent user harm, providing a scalable measurement layer for AI psychological safety.