MedMCQA
MedMCQA is a large-scale multiple-choice medical question answering benchmark dataset derived from medical entrance examinations, used to evaluate models' medical knowledge.
Timeline (last 90 days)
The study reproduced this recipe from scratch on the MedMCQA dataset and validated it on three frozen models.
On 3,842 held-out questions (random guessing accuracy is 25.0%), the two small encoders using the raw last-token vectors achieved accuracies of 33.1% and 33.0%, respectively.
After introducing the one-word summary prompt and lowering the head learning rate, the accuracies of the two small encoders rose to 41.7% and 42.5%, and Qwen3-8B reached 46.0%±0.2 (three random seeds).
Flan-PaLM and Med-PaLM 2 report accuracies of 57.6% and 72.3% on MedMCQA under different protocols, which are inconsistent with the evaluation protocol of this paper's best result of 46.0%, so direct comparison requires caution.
MedMCQA is a large-scale multiple-choice question dataset derived from Indian medical entrance exams, where each question typically provides four options and random guessing accuracy is 25%.
Medical benchmarks (such as MedQA, MedMCQA, PubMedQA, etc.) mostly use multiple-choice formats; the options themselves impose strong constraints on the answer space, and models can answer by eliminating options, pattern matching, or surface cues, so high scores may reflect test-taking ability rather than consultation competence.
All Facts (6)
The study reproduced this recipe from scratch on the MedMCQA dataset and validated it on three frozen models.
50%UnverifiedOn 3,842 held-out questions (random guessing accuracy is 25.0%), the two small encoders using the raw last-token vectors achieved accuracies of 33.1% and 33.0%, respectively.
50%UnverifiedAfter introducing the one-word summary prompt and lowering the head learning rate, the accuracies of the two small encoders rose to 41.7% and 42.5%, and Qwen3-8B reached 46.0%±0.2 (three random seeds).
50%UnverifiedFlan-PaLM and Med-PaLM 2 report accuracies of 57.6% and 72.3% on MedMCQA under different protocols, which are inconsistent with the evaluation protocol of this paper's best result of 46.0%, so direct comparison requires caution.
50%UnverifiedMedMCQA is a large-scale multiple-choice question dataset derived from Indian medical entrance exams, where each question typically provides four options and random guessing accuracy is 25%.
50%UnverifiedMedical benchmarks (such as MedQA, MedMCQA, PubMedQA, etc.) mostly use multiple-choice formats; the options themselves impose strong constraints on the answer space, and models can answer by eliminating options, pattern matching, or surface cues, so high scores may reflect test-taking ability rather than consultation competence.
50%