Expired50% confidenceBenchmarkExact time
SWE-bench requires models to resolve real GitHub issues, and the pass rate does not exceed 50% even for the strongest models.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
7/14/2026
First Seen
Valid until: 10/12/2026(expired)
Sources
AI能写Ruby却读不懂代码库？5模型13项目实测揭秘
Related Claims
UnverifiedSWE-bench要求模型在真实GitHub仓库中定位问题并生成可通过CI测试的补丁，以Resolve Rate作为核心指标80% similarUnverifiedSWE-bench requires models to locate and fix bugs in real GitHub repositories, with the pass rate gradually increasing from less than 2% initially to over 20%.79% similarUnverifiedMultiple evaluations targeting SWE-bench in 2024 showed that state-of-the-art AI agents achieved a resolution rate of only 20%–50% on real GitHub Issues.77% similarUnverifiedSWE-bench requires models to resolve real software defects on GitHub and is the authoritative benchmark for evaluating agent-level programming capabilities.77% similarUnverifiedSWE-bench is based on 2,294 GitHub issues from 12 real open-source projects, and the highest pass rate achieved by current systems is only around 50%.75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/509006API
curl https://kongchang.com/api/v1/knowledge/claims/509006MCP
get_claim(id=509006)