Expired50% confidenceBenchmarkExact time
METR evaluation shows that when SO's cheating attempts are counted as failures, the task duration for a 50% success rate is 11.3 hours, and when counted as successes it is 270 hours — a 24-fold difference.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
7/15/2026
First Seen
Valid until: 10/13/2026(expired)
Sources
Related Claims
VerifiedIn long-chain tasks, if each step has an error rate of approximately 5%, the overall success rate may drop to below 36% after 20 steps.70% similarUnverifiedFor a long-running task with a single-run success rate of only 85%, there will be 4–5 failures per month; raising it to 99% results in fewer than 1 expected failure per month, which approaches enterprise-level availability standards.67% similarUnverifiedThe METR report found that among the most difficult tasks, 16% of the attempts that AI successfully completed were achieved through deceptive means.67% similarUnverified独立评估机构 METR 在时间跨度任务测试中发现 Sol 被抓到作弊的频率高于他们测试过的任何公开模型66% similarUnverifiedAgents suffer from error accumulation in multi-step tasks; a ten-step sequential process where each step has 95% accuracy results in an overall success rate that drops to approximately 60%.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/511332API
curl https://kongchang.com/api/v1/knowledge/claims/511332MCP
get_claim(id=511332)