Unverified50% confidenceBenchmarkExact time
Across three re-runs, only 4 responses in promptfoo had their pass/fail conclusion flip.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedFinal results of the four pre-registered hypotheses: H1 (the knob is not the same tool) supported; H2 (more thinking is harmful on distractor counting) not supported; H3 (more thinking helps deductive reasoning) supported only for gpt-5.4-mini; H4 (the bill still rises when accuracy does not) supported (descriptive).62% similarUnverifiedGemma 4 26B terminated with a runtime error before producing a score, because the scoring code assumed at least one sentence would be returned in the correct structured format.60% similarUnverifiedThe author has Qwen review the generated results with a maximum of 3 retries, returning the best-matching version if all three attempts fail to meet the standard.60% similarUnverifiedAmong the three tested models, only Qwen2.5-3B can reliably refuse to answer when the evidence does not support an answer.59% similarUnverifiedQwen Next scored only 6/108 (5.56%) in testing due to format reliability issues (17 invalid_json responses, with only 6 entries scorable) rather than insufficient comprehension.58% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1018298API
curl https://kongchang.com/api/v1/knowledge/claims/1018298MCP
get_claim(id=1018298)