[KongchangAI]
Benchmark

PubMedQA

A biomedical literature question-answering benchmark used to evaluate models' ability to understand and answer questions about PubMed research abstracts.

Timeline (last 90 days)

Nov 17

Medical benchmarks (such as MedQA, MedMCQA, PubMedQA, etc.) mostly use multiple-choice formats; the options themselves impose strong constraints on the answer space, and models can answer by eliminating options, pattern matching, or surface cues, so high scores may reflect test-taking ability rather than consultation competence.

Unverified50%
Nov 6

Existing biological benchmarks mostly focus on literature question answering (such as PubMedQA) or structured bioinformatics pipelines, and struggle to reflect the ambiguity and multi-step nature of bench operations.

Unverified50%
Oct 16

Health benchmarks usually refer to evaluating a model's level of medical knowledge using standardized medical question-answering datasets (such as MedQA, PubMedQA, etc.), with accuracy as the main metric; such tests assume by default that the model should try to give an answer, but Tabibu's safety layer actively refuses diagnosis and so on, causing many questions it could have answered correctly to be intercepted by safety fallbacks.

Unverified50%

All Facts (3)

Source Articles