Unverified50% confidenceBenchmarkExact time
Before testing, the semantic threshold was fixed at 0.81, a value calibrated on the Banking77 training set to guarantee that at least 98% of semantic hits point to the same intent; but on the actual test set the intent agreement rate of semantic hits was 96.7% to 96.8%, with each model having 30 to 31 hits on the wrong intent.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/19/2026
First Seen
Valid until: 1/17/2027
Sources
Related Entities
Related Claims
UnverifiedIn a test of 119 pairs of "near-identical questions", at the 0.81 threshold 62% of modified questions incorrectly hit; even raised to 0.90, 25% still hit; the hit rate for genuine synonymous rewrites dropped from 39% to 4%.74% similarUnverifiedIn the benchmark test, the heuristic peak rule achieved 93.3% overall accuracy, 100.0% impact recall, and 38.4% cadence false positives, requiring manual threshold tuning.73% similarUnverifiedRanked by question difficulty, persistence rate was 91.4% on the 60 questions that all models answered correctly, but only 73.5% on the 52 questions that at least one model got wrong, indicating that sycophancy tracks uncertainty.73% similarUnverifiedA single "trusted source" prompt caused 7 out of 8 tested models to reverse between 45% and 88% of their correct answers.71% similarUnverifiedThis test has limitations: only 14 groups, one 2-billion-parameter model, one run; three valid paraphrases were corrected before the run because they changed the legal meaning, and the corrections are in the same commit as the test set; the test set has not yet been run on larger models.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1065405API
curl https://kongchang.com/api/v1/knowledge/claims/1065405MCP
get_claim(id=1065405)