Can Open-Source LLMs Match into a Spanish Residency? A Real MIR Exam Test

Medical exam scores are often treated as evidence of a language model's clinical usefulness, but a health system planning to run open-source models on its own hardware needs to know far more than accuracy. Can the model recognize when it might be wrong? Does medical fine-tuning improve that ability? Do correct answers rest on sound clinical reasoning? Is it actually using the images it's given? A study on the Spanish MIR residency exam set out to answer these questions, scoring 1,552 questions across eight exam sittings with four open-source models ranging from 800M to 27B parameters, using the exam's own rules.

Rankings Under Exam Rules and the "Skip" Paradox

The study's most intuitive finding: if the three larger models answered every question, their scores would roughly place them among the top 600 to 3,000 candidates, with 41 to 43 specialties still open to them. That result alone is striking—open-source models have a realistic shot at "matching into" a specialty on Spain's most demanding medical selection exam.

But the more interesting part is the skip strategy. Under exam rules, wrong answers carry a penalty, so in theory a candidate should leave questions blank when unsure. Yet the study shows skipping is never worthwhile: even the model's least confident answers have accuracy above the 25% break-even point. In other words, no matter how weak the model's "gut feeling," it beats handing in a blank sheet. This implies a systematic bias between the model's internal uncertainty signal and its true accuracy—it underestimates itself.

MIR exam scoring rules: The Spanish MIR (Médico Interno Residente) residency exam awards 3 points for a correct answer, deducts 1 point for a wrong answer, and gives zero for an unanswered question. This means the "break-even" accuracy for each question is 25%—as long as the model's probability of being correct on a question exceeds 25%, answering has positive expected value. The rule is designed to penalize blind guessing and encourage candidates to skip when unsure. The study leverages this rule to test whether the model's uncertainty signal is calibrated: if the model accurately perceives its own knowledge boundaries, it should skip when confidence is low.

Medical Fine-Tuning: Less Overconfidence, but No Better at Telling Right from Wrong

A commonly assumed causal link is that medical domain fine-tuning makes a model better at medicine, and therefore better at knowing when it doesn't know. The results throw cold water on this. Medical fine-tuning did reduce the model's overconfidence, but at the cost of lower accuracy; more importantly, it did not improve the confidence's ability to discriminate correct answers from incorrect ones.

This means fine-tuning changes the model's "posturing style," not its "metacognition." After fine-tuning, the model pounds its chest less, but its internal yardstick for distinguishing "I know this one" from "I'm guessing here" hasn't gotten any more accurate. For healthcare organizations planning to deploy open-source models, this is a warning sign: the drop in confidence from fine-tuning could be misread as improved reliability, when in fact it's just conservatism.

Metacognition and confidence calibration: In machine learning, a model's "metacognition" typically manifests as confidence calibration—whether the confidence it outputs truly reflects the probability that its answer is correct. A well-calibrated model that says "I'm 80% sure" should actually be right close to 80% of the time. Discrimination refers to whether confidence can separate correct from incorrect answers, commonly measured by AUROC. Medical fine-tuning may shift the model's output distribution (e.g., lowering confidence overall) without necessarily improving discrimination. The study's results show that the fine-tuned model is merely "more humble," not "more self-aware."

Correct Answers Don't Mean Correct Clinical Reasoning

The finding with the most clinical significance is the "quality" of correct answers. In 13% to 38% of correct responses, the model's reasons for ruling out distractors were clinically untenable—and confidence didn't reflect this at all.

In other words, the model got it right, but its path to the right answer was wrong. It may have eliminated wrong options through literal word associations, superficial co-occurrences in training data, or some heuristic unrelated to clinical practice. In real clinical settings, this "right answer, wrong reasoning" is extremely dangerous: the same reasoning style applied to a different question or a different patient could lead to the wrong conclusion. And the model has no awareness of this—its confidence remains steady.

Images Are Effectively Decorative

The study also tested whether the model actually uses the context images provided. The result was clear-cut: removing the image or swapping it out left accuracy unchanged.

This shows that image information is effectively ignored when these open-source models handle MIR questions. For clinical scenarios where medical imaging is a key diagnostic basis, this is a direct usability warning—the model may appear to "look at" the image while answering, but in reality it's taking shortcuts based on textual cues. The existence of multimodal capability does not mean multimodal information is effectively used.

The trap of multimodal medical benchmarks: Many multimodal benchmarks evaluate models solely on final-answer accuracy, ignoring whether the model actually relies on the image. A common problem is the "text shortcut"—the question text already contains enough clues for the model to answer correctly without reading the image, leading to an overestimation of multimodal capability. Rigorous evaluation should include image ablation experiments (removing the image) and image substitution experiments (replacing it with an irrelevant image); if accuracy is unchanged, the model is proven not to be using visual information. This study adopted this approach, revealing the "pseudo-multimodal" phenomenon of open-source models on MIR questions.

Implications for Deploying Open-Source Medical Models

The picture this study paints isn't pessimistic, but it is sobering enough. Open-source models now perform at a level close to "matching" on standardized medical exams, but the gap between exam scores and clinical usability lies precisely beyond the accuracy numbers: uncertainty signals are unreliable, skip strategies fail, fine-tuning doesn't improve metacognition, correct answers may rest on flawed clinical reasoning, and image input is effectively ignored.

For a health system planning to run open-source models on its own hardware, these findings point to a pragmatic conclusion: pre-deployment evaluation cannot look at accuracy alone. It must specifically test how the model behaves when uncertain, the clinical soundness of its reasoning chains, and whether multimodal input is genuinely being used. Otherwise, a model that "matches into a residency" could make well-reasoned but wrong-headed judgments at the bedside.

One-line Summary

Open-source models can pass the Spanish MIR exam, but their uncertainty signals, reasoning quality, and image use are all unreliable.

Paragraph Summary

A study on the Spanish MIR residency exam scored 1,552 questions across eight sittings with four open-source models (800M to 27B parameters), finding that the three larger models would rank among the top 600 to 3,000. Skipping was never worthwhile, indicating the models underestimate their own accuracy. Medical fine-tuning reduced overconfidence but sacrificed accuracy and did not improve confidence's ability to discriminate right from wrong. In 13% to 38% of correct answers, distractors were ruled out on clinically untenable grounds, with confidence reflecting none of it. Accuracy was unchanged when images were removed or replaced, showing the models didn't actually use them. The study shows a gap between exam scores and clinical usability, and that pre-deployment testing must specifically probe uncertainty behavior, reasoning soundness, and multimodal information use.