MedAbstain
A medical question answering evaluation benchmark focused on abstention behavior, emphasizing assessment of a model's expression of not knowing in response to questions beyond its knowledge boundary.
Timeline (last 90 days)
The evaluation used two benchmarks: MedAbstain and ClinDet-Bench.
On the MedAbstain benchmark, the median appropriate abstention rate of the base models was only 8.3%, and after fine-tuning it further dropped to a median of 4.2%.
When deploying biomedical large models, it is recommended to test the change in the model's abstention rate before and after fine-tuning using a benchmark similar to MedAbstain, in order to avoid erosion of the safety guardrails.
All Facts (3)
The evaluation used two benchmarks: MedAbstain and ClinDet-Bench.
50%UnverifiedOn the MedAbstain benchmark, the median appropriate abstention rate of the base models was only 8.3%, and after fine-tuning it further dropped to a median of 4.2%.
50%UnverifiedWhen deploying biomedical large models, it is recommended to test the change in the model's abstention rate before and after fine-tuning using a benchmark similar to MedAbstain, in order to avoid erosion of the safety guardrails.
50%