MedQA
An AI evaluation benchmark for clinical medical question answering, incorporating licensed physicians in annotation and review processes. It assesses model performance in medical knowledge and clinical decision-making scenarios.
Timeline (last 90 days)
Health benchmarks usually refer to evaluating a model's level of medical knowledge using standardized medical question-answering datasets (such as MedQA, PubMedQA, etc.), with accuracy as the main metric; such tests assume by default that the model should try to give an answer, but Tabibu's safety layer actively refuses diagnosis and so on, causing many questions it could have answered correctly to be intercepted by safety fallbacks.
COMED achieves an improvement of up to +10.7 percentage points on the MedQA medical question-answering benchmark.
MedQA is a multiple-choice question-answering benchmark constructed from the United States Medical Licensing Examination (USMLE) question bank.
Both MedQA and MultiMedBench involve licensed physicians in annotation and review, setting a precedent for the expert-informed design pattern in medical AI evaluation.
All Facts (4)
Health benchmarks usually refer to evaluating a model's level of medical knowledge using standardized medical question-answering datasets (such as MedQA, PubMedQA, etc.), with accuracy as the main metric; such tests assume by default that the model should try to give an answer, but Tabibu's safety layer actively refuses diagnosis and so on, causing many questions it could have answered correctly to be intercepted by safety fallbacks.
50%UnverifiedCOMED achieves an improvement of up to +10.7 percentage points on the MedQA medical question-answering benchmark.
50%UnverifiedMedQA is a multiple-choice question-answering benchmark constructed from the United States Medical Licensing Examination (USMLE) question bank.
50%UnverifiedBoth MedQA and MultiMedBench involve licensed physicians in annotation and review, setting a precedent for the expert-informed design pattern in medical AI evaluation.
50%