[KongchangAI]
Benchmark

MedMCQA

MedMCQA is a large-scale multiple-choice medical question answering benchmark dataset derived from medical entrance examinations, used to evaluate models' medical knowledge.

Timeline (last 90 days)

Nov 19

The study reproduced this recipe from scratch on the MedMCQA dataset and validated it on three frozen models.

Unverified50%
Nov 19

On 3,842 held-out questions (random guessing accuracy is 25.0%), the two small encoders using the raw last-token vectors achieved accuracies of 33.1% and 33.0%, respectively.

Unverified50%
Nov 19

After introducing the one-word summary prompt and lowering the head learning rate, the accuracies of the two small encoders rose to 41.7% and 42.5%, and Qwen3-8B reached 46.0%±0.2 (three random seeds).

Unverified50%
Nov 19

Flan-PaLM and Med-PaLM 2 report accuracies of 57.6% and 72.3% on MedMCQA under different protocols, which are inconsistent with the evaluation protocol of this paper's best result of 46.0%, so direct comparison requires caution.

Unverified50%
Nov 19

MedMCQA is a large-scale multiple-choice question dataset derived from Indian medical entrance exams, where each question typically provides four options and random guessing accuracy is 25%.

Unverified50%
Nov 17

Medical benchmarks (such as MedQA, MedMCQA, PubMedQA, etc.) mostly use multiple-choice formats; the options themselves impose strong constraints on the answer space, and models can answer by eliminating options, pattern matching, or surface cues, so high scores may reflect test-taking ability rather than consultation competence.

Unverified50%

All Facts (6)

Source Articles