Unverified50% confidenceFactExact time
On 605 English questions, the released checkpoint's answer flip rate was 19.7%.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/9/2026
First Seen
Valid until: 2/7/2027
Sources
Related Entities
Related Claims
UnverifiedThe released checkpoint changes its answer due to cyclic rotation of options on 25.8% of questions (confidence interval 22.9-28.7%).74% similarUnverifiedIn a test of 119 pairs of "near-identical questions", at the 0.81 threshold 62% of modified questions incorrectly hit; even raised to 0.90, 25% still hit; the hit rate for genuine synonymous rewrites dropped from 39% to 4%.69% similarUnverifiedOn the consensus reference standard formed after review, another annotator's agreement with that consensus on 200 randomly sampled tweets was 60.5% (κ=0.517).67% similarUnverifiedAt the 0.81 threshold, the proportion of cached answers that "fit the question" is 5.8 to 10.7 percentage points lower than fresh answers; converted to all responses, the overall quality loss is about 2 to 3 percentage points; Haiku 5.5 has the largest gap (10.7 percentage points), with a cached-answer pass rate of only 67.9%.67% similarUnverifiedWhen asked directly, the accuracy of the decision models falls in the 50.0%–67.0% range.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1149597API
curl https://kongchang.com/api/v1/knowledge/claims/1149597MCP
get_claim(id=1149597)