[KongchangAI]
BenchmarkMedAbstain

MedAbstain

A medical question answering evaluation benchmark focused on abstention behavior, emphasizing assessment of a model's expression of not knowing in response to questions beyond its knowledge boundary.

Timeline (last 90 days)

Nov 9

The evaluation used two benchmarks: MedAbstain and ClinDet-Bench.

Unverified50%
Nov 9

On the MedAbstain benchmark, the median appropriate abstention rate of the base models was only 8.3%, and after fine-tuning it further dropped to a median of 4.2%.

Unverified50%
Nov 9

When deploying biomedical large models, it is recommended to test the change in the model's abstention rate before and after fine-tuning using a benchmark similar to MedAbstain, in order to avoid erosion of the safety guardrails.

Unverified50%

All Facts (3)

Source Articles