Medical Fine-Tuning Reshapes Rather Than Reduces Risk: A New Risk-Sensitive Evaluation Framework for AI Patient Q&A

Medical large language models are moving from research demos to production deployment, yet evaluation still centers on factual correctness. Factual correctness is necessary but not sufficient—a response can be factually accurate yet harm patients by giving dangerous instructions. To address this, researchers propose a risk-sensitive evaluation framework that scores six categories of risk language (treatment instructions, contraindications, dosage expressions, emergency triage prompts, high-alert medication mentions, and overconfident assertions) along with their severity, anchored to published clinical harm taxonomies. Severity is classified by an ensemble of three LLM judges. The framework is applied to six 7-9B instruction-tuned models (three general-purpose: Mistral-7B, Phi-3.5-mini, Llama-3.1-8B; three medical-adapted: BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), tested on 200 real patient questions from the K-QA benchmark.

Key Finding: Factual Hallucinations and Risk Language Are Mutually Independent

Risk-language ensemble scores and factual hallucination rates are statistically independent (pooled Spearman ρ = -0.065, n = 300, 95% CI [-0.18, +0.05]; distance correlation = 0.10, independence test p = 0.24). This means a model may have high factual accuracy yet frequently output high-risk instructions, and vice versa. Factuality evaluation alone cannot capture patient safety risks.

Spearman's rank correlation coefficient is a non-parametric statistic that measures the monotonic relationship (not necessarily linear) between two variables, ranging from -1 to +1. A ρ near 0 indicates no monotonic association. Distance correlation, by contrast, can detect both linear and nonlinear dependencies, ranging from 0 to 1, with 0 indicating independence. Using both together avoids missing nonlinear associations that would be overlooked under a linearity assumption alone. Both metrics are reported here to argue more robustly that there is no substantive association between risk-language scores and factual hallucination rates.

Medical Fine-Tuning: Reshaping Risk Rather Than Reducing It

Medical fine-tuning did not reduce overall risk but changed its composition:

• Continued pre-training adaptation (BioMistral, OpenBioLLM) consistently suppressed high-alert medication mentions while amplifying overconfident assertions.
• Clinical instruction tuning (Med42) suppressed only dosage expressions without amplifying overconfidence.

This finding challenges the intuition that medical fine-tuning is necessarily safer. Different fine-tuning strategies have heterogeneous effects on risk categories, requiring targeted evaluation.

Continued pre-training refers to continuing self-supervised training of a general base model on domain corpora (e.g., medical literature, clinical guidelines) to inject domain knowledge, without aligning to specific instructions. Instruction tuning, by contrast, occurs after pre-training and trains the model on instruction-response pairs to follow instructions and complete specific tasks. In this work, BioMistral and OpenBioLLM use continued pre-training adaptation, while Med42 uses clinical instruction tuning, explaining the heterogeneous composition of risk categories across the two model types.

Evaluation Reliability: Language-Anchored Categories Reliable, Clinical-Judgment Categories Questionable

Category-level Krippendorff's α shows that the four language-anchored categories (treatment instructions, contraindications, dosage expressions, high-alert medication mentions) have α between 0.53 and 0.71, ranging from fair to good, whereas the two categories relying on clinical judgment (emergency triage prompts, overconfident assertions) have α of only 0.19 and 0.31, indicating low reliability. This suggests automated safety evaluation is relatively reliable for linguistically explicit risks but remains unreliable for risks requiring clinical reasoning.

Krippendorff's α is a statistic measuring agreement among multiple coders (or judges), applicable to any number of judges and data types. Its values typically range from 0 to 1, with α ≥ 0.80 generally considered reliable, 0.67–0.80 acceptable but requiring caution, and below 0.67 indicating insufficient agreement. Here, language-anchored categories have α between 0.53 and 0.71, at a "fair" level, while clinical-judgment categories have α of only 0.19 and 0.31, far below common thresholds, indicating that automated identification results for these categories are unreliable.

Framework and Code Open-Sourced

The researchers released the scoring rubric, judge prompts, and evaluation code, providing the community with a reproducible risk assessment tool. The framework is expected to complement existing factuality benchmarks and advance the safe deployment of patient-facing AI.

One-line Summary

Medical LLMs need risk-sensitive evaluation: risk language is independent of factual hallucinations, and fine-tuning reshapes rather than reduces risk.

Paragraph Summary

The article argues that evaluating medical LLMs cannot rely on factual correctness alone, because factually accurate responses may still contain dangerous instructions. Researchers propose a risk-sensitive evaluation framework that scores six categories of risk language and their severity across six 7-9B models. The key finding is that risk-language scores and factual hallucination rates are statistically independent, so factuality evaluation alone cannot capture patient safety risks. Medical fine-tuning does not reduce overall risk but changes its composition: continued pre-training adaptation suppresses high-alert medication mentions but amplifies overconfidence, while clinical instruction tuning suppresses only dosage expressions. Regarding evaluation reliability, language-anchored categories show fair-to-good agreement, whereas clinical-judgment categories have low reliability. The researchers open-sourced the scoring rubric, judge prompts, and evaluation code.