Unverified50% confidenceBenchmarkExact time
In the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/17/2026
First Seen
Valid until: 2/15/2027
Sources
Related Entities
Related Claims
UnverifiedA single performance percentage is misleading; a model that claims to retain 98% of performance may happen to fall within the lost 2% for a specific task type, and real tasks should be tested before trusting benchmarks.70% similarUnverifiedReplay-based validation can detect 76.2% of unsuccessful improvements, whereas build-check-only approaches fail to detect any of these behavioral failures.69% similarUnverifiedIn the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.69% similarUnverifiedThe best-performing largest model can solve 79.6% of held-out test set problems, but only 55.3% on TranSGrid, dropping sharply to 15.8% on the hardest subset.69% similarUnverifiedThe user pointed out that these models achieve high scores on benchmarks yet frequently fail at the simplest tasks.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1179746API
curl https://kongchang.com/api/v1/knowledge/claims/1179746MCP
get_claim(id=1179746)