Unverified50% confidenceBenchmarkExact time
Claude Haiku 4.5 answered 'not found' in 21 of its 24 errors on two-step questions, while GPT-5.4 mini copied the literal wording of the referral as the answer in 20 of its 27 errors, and fabricated real-sounding names while citing actual documents in another 7.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/12/2026
First Seen
Valid until: 1/10/2027
Sources
Related Entities
Related Claims
UnverifiedIn the hard-to-verify category with injection bait, the number of questions for which GPT-5.4 mini judged incorrectly in both orderings rose from 3 to 39, Qwen3-Next 80B Instruct from 4 to 38, and GPT-5.4 mini and Qwen3-Next Instruct scored zero on hard questions (below the approximately 0.25 expected from random coin flipping).70% similarUnverifiedClaude Opus 4.6 produced 30 times more errors than GPT-5.4, averaging 2.5 errors per trajectory vs. 0.08, with the difference being statistically significant (p < 0.0001, Cliff's δ = 0.613).67% similarUnverifiedGPT-5.5 ran out of usage quota, leaving 35 of the 90 questions unfinished; all 55 answered questions were correct, but Kaggle counted the unanswered questions as wrong, causing its official score to be lower.64% similarUnverified46个模型中有41个完整跑完全部28道题，3个因服务错误未完成，GPT-5.6 Terra因配额限制止步24/28，GPT-6 Astra只跑了1道题64% similarUnverifiedOn two-step reasoning questions, Claude Haiku 4.5 scored 6/30, GPT-5.4 mini scored 3/30, while Claude Sonnet 5.5, Gemini 3.8 Flash, and Gemma 4 all scored 30/30.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1003221API
curl https://kongchang.com/api/v1/knowledge/claims/1003221MCP
get_claim(id=1003221)