Unverified50% confidenceBenchmarkExact time
Models that only read the call transcripts passed 29 of the 31 contradictory cases.
1
Sources
50%
Confidence
Long-term
Relevance
9/25/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA hand-crafted library of 22 calculators achieved perfect accuracy on the 440 cases it supported, but abstained on the remaining cases, yielding an overall coverage rate of only 40.0%.63% similarUnverifiedAmong the 35 genuine failure cases, only 3 did not involve any assertions, indicating that the vast majority of failures stem from accepting biased statements rather than from missing information.62% similarUnverifiedEven the best-performing models can only recover 30% of the evaluated evidence and 25% of the limitations descriptions.61% similarUnverifiedIn the most difficult single-concept quantitative category, most models' accuracy falls below 30%61% similarUnverifiedAIME has only 30 samples, and tiny score differences derived from running a small number of seeds are not statistically significant.59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/949211API
curl https://kongchang.com/api/v1/knowledge/claims/949211MCP
get_claim(id=949211)