Unverified50% confidenceFactExact time
In the corrected data, counted by change, 14 of 91 agent PRs contain WEAK tests, while only 1 of 71 maintainer commits does.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedBy re-tallying the results files, readers found a row of data that the authors had skipped: the maintainer file lists 9 of 212 tests judged WEAK (4%), while the agent file lists 79 of 591 (13%).73% similarUnverifiedThe current dataset contains 770 recorded attempts across 29 model/configuration entries, but the unified test suite includes only 16 distinct tasks.64% similarUnverifiedThe research team conducted a retrospective ablation analysis across 12 models with a total of 121,680 valid trials, of which 79,853 failed the executability check.64% similarUnverifiedThe test suite consists of 10 benchmarks; most run 1,000 questions each, with HumanEval comprising 164 questions and TruthfulQA comprising 817 questions.63% similarUnverifiedThe researchers confirmed 91 failures through manual verification and categorized them into 7 failure loci.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1028488API
curl https://kongchang.com/api/v1/knowledge/claims/1028488MCP
get_claim(id=1028488)