Unverified50% confidenceBenchmarkExact time
On the public JevBench benchmark (231 downloadable questions), the unmodified 4B version scored 170/231 overall, with 48/48 correct in the Easy group, 69/72 in the Original group, and only 53/111 in the Hard group, with 58 of the 61 errors coming from the Hard group.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/18/2026
First Seen
Valid until: 11/1/2026
Sources
Related Entities
Related Claims
UnverifiedThere are 33 technical errors in the dataset; the largest cluster is the Qwen 3.5 35B-A3B configuration, with 22 errors out of 32 attempts.69% similarUnverifiedIn a test of 48 synthesized programming requests, under the default four-option question Laya matched the team's annotation on only 21 of them, with a maximum confidence of only 0.26.65% similarUnverifiedClaude Haiku 4.5 answered 'not found' in 21 of its 24 errors on two-step questions, while GPT-5.4 mini copied the literal wording of the referral as the answer in 20 of its 27 errors, and fabricated real-sounding names while citing actual documents in another 7.63% similarUnverifiedIn project three (Lakebed product audit), Gemini 4 Argon scored only 2.25/10, with 56 of 78 findings being wrong, a failure rate of about 70%; while 6.1 Sol scored 9/10.62% similarUnverifiedMistral Small 4 scored only 0.699 on unit conversion inputs and produced 265 errors that differed from the correct label by exactly a power of ten.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1064023API
curl https://kongchang.com/api/v1/knowledge/claims/1064023MCP
get_claim(id=1064023)