Unverified50% confidenceBenchmarkExact time
When the open-source small model gemma2:2b was used as the checker, it accepted 8 of the 28 wrong answers (6 wrong party and 2 wrong action), while rejecting 10 of the 14 correct answers.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
UnverifiedIn tests on 42 candidate answers about Romanian legal deadlines, the author's own AI safety check system accepted all 28 wrong answers (14 wrong party and 14 wrong action), while rejecting 6 of the 14 correct answers.75% similarUnverifiedIn a verbatim-quotation benchmark spanning 13 models, 2 instruction variants, and 962 scored responses, not a single model corrected any planted error within a quotation.75% similarUnverifiedEach model answered 112 questions (7 flaw types × 2 versions × 4 repetitions × 2 conditions), with half flawed and half sound, and a model that mechanically answered 'none' would receive a baseline score of 0.50.73% similarUnverifiedRanked by question difficulty, persistence rate was 91.4% on the 60 questions that all models answered correctly, but only 73.5% on the 52 questions that at least one model got wrong, indicating that sycophancy tracks uncertainty.73% similarUnverifiedAfter adding the validation rules, the developer added 7 untuned questions, and of the 31 questions run online, 28 were correct; one of the failures came from the developer's own bug (the model output '$8.00' while the website wrote '$8', and number validation judged it a mismatch), and after the fix, the same draft passed 29/31.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1045677API
curl https://kongchang.com/api/v1/knowledge/claims/1045677MCP
get_claim(id=1045677)