Unverified50% confidenceBenchmarkExact time
In the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/17/2026
First Seen
Valid until: 2/15/2027
Sources
Related Entities
Related Claims
UnverifiedIn the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.69% similarUnverifiedOn a test subset of 152 completely untouched questions, anchored hybrid retrieval achieved 42.11%, while the planner-guided method reached only 36.84%.68% similarUnverifiedTraditional benchmarks have become ineffective for evaluating model reasoning ability because almost every model may have seen the standard answers in its training data.65% similarUnverifiedIn the initial workup stage and the post-treatment follow-up stage, the dominant failure mode is decision omission, accounting for 33.4% and 30.4%, respectively.65% similarUnverifiedThe original video cited a benchmark test: the model failed to complete a task when given 46 tools, yet passed the same task once the tools were reduced to 19.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1179750API
curl https://kongchang.com/api/v1/knowledge/claims/1179750MCP
get_claim(id=1179750)