MedPhysBench: A New Benchmark for Medical Physics Calculations and Error Detection in Language Models

Medical physicists work with an extremely low tolerance for error—a single mistake in beam calibration or monitor unit (MU) calculation could affect every patient treated on that equipment. To systematically evaluate the reliability of language models in these high-stakes calculations, researchers have introduced the MedPhysBench benchmark, covering four major domains: radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection. The hardest tasks reach clinical levels such as TG-51 calibration worksheets.

Calculation and Error Detection: The Benchmark's Dual Design

MedPhysBench contains ten seeded generators used to produce calculation problems; reference answers are generated by code implementing the relevant protocols, and a grader handles unit conversions and applies tolerance intervals. The same batch of inputs also derives independent "check" problems: the model is shown a colleague's complete worksheet, or only the colleague's final result, with half correct and half containing a single planted error, to test the model's ability to spot mistakes.

This dual-track design mirrors real clinical scenarios—physicists must both calculate correctly themselves and be able to catch colleagues' errors.

Monitor Units (MU) and Inverse Square Correction

Monitor Unit (MU) is the unit of output dose from a linear accelerator. Physicists must convert parameters such as prescribed dose, tissue depth, field size, and source-to-skin distance into the MU setting. The inverse square correction is one of the core steps: when the treatment distance deviates from calibration conditions, the dose rate changes inversely with the square of the distance, requiring a factor such as $(\text{SSD}_{\text{cal}}/\text{SSD})^2$ in the formula. Omitting this term systematically overestimates or underestimates dose—a typical and dangerous clinical calculation error, which is precisely why it is used as a targeted test case in the benchmark.

TG-51 Protocol

TG-51 is the reference dosimetry protocol for high-energy photon and electron beams from medical accelerators, published by the American Association of Physicists in Medicine (AAPM). It specifies the standard procedure for calibrating absorbed dose to water for high-energy photon and electron beams, including measurement equipment, measurement depth, beam quality correction factor $k_Q$, and temperature-pressure corrections. TG-51-based calibration worksheets are a core daily tool for clinical physicists, and incorporating them into the benchmark means evaluation tasks must cover multi-step table lookups, unit conversions, and chained factor multiplications.

NCRP Report 151

NCRP Report 151 is the National Council on Radiation Protection and Measurements report on structural shielding design for external beam radiation therapy. It provides parameters such as workload, use factor, and occupancy factor, along with rules for their values, needed to calculate shielding thickness. Because the report permits multiple compliant approaches in some situations, a model answering according to the report may disagree with the reference answer if the problem does not explicitly constrain the assumptions—this is exactly the source of several "marked wrong but actually compliant" answers in the benchmark.

Calculation Performance of 12 Models

On a held-out test set of 1,200 problems (half of which were used for two models), the 12 models' calculation scores ranged from 33.3% to 100.0%. The four top-scoring models (99.3%–100.0%) had a total of 9 answers marked wrong, of which 5 were the models' own errors and 4 followed the provisions of NCRP Report 151 where the benchmark problem was incorrectly stated or left open on the relevant point.

Lower-ranked models generally performed worst on level-3 clinical tasks. One typical failure mode: Claude Sonnet 5.5 omitted the inverse square correction in 19 of 20 isocentric MU problems.

Error Detection Ability Varies Markedly

In the error-detection tasks, five models detected at least 96.7% of planted errors (mostly obvious ones) with a false-positive rate of no more than 4%. Of the remaining seven models, four showed a significant increase in false positives when only the colleague's result was provided without the worksheet.

An exploratory targeted test used 20 public "omitted inverse square correction" cases. The three best-performing models caught the error in every worksheet; three weaker models accepted 16 to 18 of the 20 worksheets with planted errors, and one of them accepted a colleague's erroneous worksheet despite applying the correction in most of its own calculations. Claude Opus 5.5 accepted 6 of the 20 erroneous results, even though it correctly applied the correction in all 20 of its own calculations.

Why False-Positive Rates Matter in Error-Detection Tasks

When only a colleague's final result is shown without the worksheet, the model lacks intermediate steps as corroboration and must rely on an independent judgment of the result's plausibility. A rise in false positives means the model tends to flag correct results as wrong, which is equally harmful in clinical review scenarios—excessive false positives consume physicists' review effort and erode trust in assistive tools. Therefore, evaluating error-detection ability requires examining both detection rate and false-positive rate; looking at detection rate alone overestimates usability.

Implications for Clinical Deployment

The benchmark reveals a key asymmetry: being able to calculate does not mean being able to check. Some models correctly apply a given correction when calculating on their own, yet let the same type of error pass when reviewing others' work—a direct challenge to the safety assumption of using language models as calculation review assistants.

The researchers also reported two issues in shielding problems discovered after the run concluded, and open-sourced the code, per-problem results, and targeted test responses to facilitate community review and improvement.

Frequently Asked Questions

What domains does MedPhysBench cover? Radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection, including clinical-level tasks such as TG-51 calibration worksheets.

Does a low score necessarily mean a model is bad? Not necessarily. Some answers marked wrong actually comply with NCRP Report 151, exposing flaws in how the problems themselves are stated.

Can models reliably review colleagues' calculations? Not across the board at present. Strong and weak models differ markedly in error detection rate and false-positive rate, and there is the phenomenon of "calculating correctly oneself but letting others' errors pass."

One-line Summary

MedPhysBench evaluates the reliability of 12 language models in clinical-grade medical physics calculations and error detection, revealing that "being able to calculate is not the same as being able to check."

Paragraph Summary

MedPhysBench is a benchmark for high-stakes calculations in medical physics, covering radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection, with the hardest tasks reaching the level of TG-51 calibration worksheets. The benchmark uses ten seeded generators to produce calculation problems, with reference answers generated by code implementing the relevant protocols; the same inputs also derive "check" problems with planted errors to test models' ability to review colleagues' worksheets. On a 1,200-problem held-out test set, the 12 models' calculation scores ranged from 33.3% to 100.0%, with lower-ranked models performing worst on level-3 clinical tasks—a typical failure being Claude Sonnet 5.5 omitting the inverse square correction in 19 of 20 isocentric MU problems. Error-detection tasks show even greater variation: five models detected at least 96.7% of planted errors with a false-positive rate of no more than 4%, while other models showed significantly higher false positives when given only results; Claude Opus 5.5, despite calculating all its own answers correctly, accepted 6 of 20 erroneous results, highlighting the asymmetry between "being able to calculate" and "being able to check."