SWE-Bench
SWE-Bench is a benchmark dataset constructed by academic researchers for evaluating the software engineering capabilities of AI systems. It collects GitHub issues and corresponding code repair tasks from real open-source repositories, requiring models to automatically generate code patches that pass test cases based on problem descriptions. The benchmark measures an AI system's ability to localize issues, understand code, and perform repairs in real-world software engineering scenarios.
Core Facts
SWE-bench directly extracts issues and corresponding pull requests from real GitHub open-source projects, requiring AI systems to understand problem descriptions, locate code defects, and generate correct fix patches
SWE-Bench was released by a research team at Princeton University in 2023.
SWE-bench was developed by a Princeton University team; the Pro version requires models to locate bugs and generate fix patches within real GitHub repositories.
Timeline (last 90 days)
Most mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
Current mainstream Agent evaluation approaches fall into three categories: task completion rate based on deterministic assertions, LLM-as-Judge, and end-to-end benchmark test sets (such as SWE-Bench and WebArena).
All Facts (20)
SWE-bench directly extracts issues and corresponding pull requests from real GitHub open-source projects, requiring AI systems to understand problem descriptions, locate code defects, and generate correct fix patches
90%VerifiedSWE-Bench was released by a research team at Princeton University in 2023.
90%VerifiedSWE-bench was developed by a Princeton University team; the Pro version requires models to locate bugs and generate fix patches within real GitHub repositories.
80%VerifiedSWE-bench was released in 2023 by a research team at Princeton University, collecting 2,294 real GitHub Issues and their corresponding Pull Request fixes from 12 well-known Python open-source projects.
80%VerifiedSWE-Bench collected 2,294 real Issue–Pull Request pairs from 12 mainstream Python open-source projects on GitHub.
80%VerifiedSWE-bench is a software engineering capability evaluation benchmark introduced by a Princeton University research team in 2023.
80%VerifiedSWE-bench (the real GitHub Issue fix rate) is regarded by the industry as the most valuable metric for measuring coding Agent capabilities.
80%VerifiedMainstream benchmarks for evaluating programming large models include HumanEval, MBPP, SWE-bench, and LiveCodeBench
80%VerifiedSWE-Bench was introduced by Princeton University in 2023 and evaluates AI programming ability by drawing thousands of resolved issues from real open-source projects on GitHub.
80%VerifiedSWE-bench was proposed by Princeton University and contains 2,294 software engineering problems from real GitHub repositories.
80%VerifiedIn early 2024, the strongest AI models scored only in the 10%–20% range on SWE-Bench.
65%VerifiedThe training data sources for specialized programming models include open-source code from GitHub, competitive programming solutions, technical documentation, and code review records, and the models are optimized for benchmarks such as HumanEval, MBPP, and SWE-bench.
65%VerifiedDevin resolved 13.86% of real GitHub Issues in the SWE-bench test, surpassing the performance of Claude 2.
65%VerifiedSWE-Bench was released by a Princeton University research team and collects real issue reports and their corresponding pull request fix patches from GitHub.
65%Partially VerifiedThe SWE-Bench test set contains 2,294 real bug-fix tasks from 12 mainstream Python open-source projects.
80%Unverifiedo3-mini-high在SWE-Bench上修复了39%的错误，允许调用内部工具后成功率飙升到61%
80%Unverified2024年初SWE-Bench最好的系统得分仅在30%左右
70%Unverified在SWE-Bench等真实软件工程基准中，顶尖模型往往需要平均30-50次工具调用才能完成一个Issue修复
70%Partially VerifiedProjects collected by SWE-bench include well-known Python open-source projects such as Django, scikit-learn, and sympy
65%UnverifiedMost mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
50%