Unverified50% confidenceBenchmarkExact time
Flan-PaLM and Med-PaLM 2 report accuracies of 57.6% and 72.3% on MedMCQA under different protocols, which are inconsistent with the evaluation protocol of this paper's best result of 46.0%, so direct comparison requires caution.
1
Sources
50%
Confidence
Long-term
Relevance
11/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA gating accuracy of 58% reported in the paper was mistakenly treated as end-to-end quality, whereas the true end-to-end quality is 98%.72% similarUnverified研究表明人类评估者在判断LLM输出的事实准确性时,正确率往往只有50%-70%69% similarUnverifiedStrict completion was 17/72 under both conditions, and verdict accuracy went from 47/72 to 46/72, remaining almost unchanged.68% similarUnverifiedOn a 50-sample evaluation set, an accuracy change from 74% to 76% corresponds to just one more correct sample, a difference that is statistically almost meaningless.68% similarUnverifiedThe confidence reported by the model is severely disconnected from actual accuracy: the 95% confidence tier had an actual accuracy of 0%, while the 90% and 70% tiers both had an actual accuracy of 100%.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1186764API
curl https://kongchang.com/api/v1/knowledge/claims/1186764MCP
get_claim(id=1186764)