MedPhysBench
A benchmark for evaluating the reliability of language models on medical physics calculation and error-detection tasks, covering four domains: radiation therapy, diagnostic imaging, nuclear medicine and radiation protection.
Timeline (last 90 days)
MedPhysBench is a benchmark for systematically evaluating the reliability of language models on medical physics calculation and error-correction tasks, covering four major domains: radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection.
MedPhysBench contains ten seeded generators for producing calculation problems, reference answers are generated by code implementing the relevant protocols, and the scorer handles unit conversion and applies tolerance intervals.
MedPhysBench's error-correction task shows the model a colleague's full worksheet or only the final result, with half correct and half containing a single implanted error, thereby testing the model's ability to find errors.
On a held-out test set of 1,200 problems, the calculation scores of 12 models range from 33.3% to 100.0%.
In the error-correction task, five models detect at least 96.7% of implanted errors, with a false-positive rate of no more than 4%.
In 20 public targeted test cases of "omitted inverse-square correction," the three best-performing models caught the error in every worksheet, while the three weaker models accepted 16 to 18 of the 20 worksheets with implanted errors.
The researchers open-sourced MedPhysBench's code, per-problem results, and targeted test responses.
All Facts (7)
MedPhysBench is a benchmark for systematically evaluating the reliability of language models on medical physics calculation and error-correction tasks, covering four major domains: radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection.
50%UnverifiedMedPhysBench contains ten seeded generators for producing calculation problems, reference answers are generated by code implementing the relevant protocols, and the scorer handles unit conversion and applies tolerance intervals.
50%UnverifiedMedPhysBench's error-correction task shows the model a colleague's full worksheet or only the final result, with half correct and half containing a single implanted error, thereby testing the model's ability to find errors.
50%UnverifiedOn a held-out test set of 1,200 problems, the calculation scores of 12 models range from 33.3% to 100.0%.
50%UnverifiedIn the error-correction task, five models detect at least 96.7% of implanted errors, with a false-positive rate of no more than 4%.
50%UnverifiedIn 20 public targeted test cases of "omitted inverse-square correction," the three best-performing models caught the error in every worksheet, while the three weaker models accepted 16 to 18 of the 20 worksheets with implanted errors.
50%UnverifiedThe researchers open-sourced MedPhysBench's code, per-problem results, and targeted test responses.
50%