Unverified50% confidenceFactExact time
In the six-person study sample, there were only about 23 moments where the tutor needed to decide 'what help to give', which is too small a sample to judge the quality of the model's decisions; by contrast, 123 unit tests can be written for the pure functions.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe test sample size is relatively limited (5 models, 13 codebases), and the conclusions await validation at a larger scale.72% similarUnverifiedIn a test covering 8 local models and more than 5000 calls, the wording style of the prompt (plain/normal/florid) had almost no effect on accuracy on non-math tasks, with differences of no more than 4 percentage points for models of 8B and above.72% similarUnverifiedThe sample size of this comparison experiment is limited: each question was asked twice, so the actual number of distinct questions covered is only about 14, and a single correct or incorrect answer greatly perturbs the conclusion.72% similarUnverifiedIn a test of 12 models across 112 questions, when faced with a "Are you sure?" challenge, models had a 94.4% probability of maintaining their original correct answer.72% similarUnverifiedThis test has limitations: only 14 groups, one 2-billion-parameter model, one run; three valid paraphrases were corrected before the run because they changed the legal meaning, and the corrections are in the same commit as the test set; the test set has not yet been run on larger models.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069221API
curl https://kongchang.com/api/v1/knowledge/claims/1069221MCP
get_claim(id=1069221)