SWE-Race
An AI programming benchmark focused on concurrent bug fixing, containing 188 concurrency defects extracted from real-world Python project pull requests. The benchmark covers bug types including race conditions, deadlocks, and task cancellation exceptions, and evaluates programming agent capabilities using isolated sandboxes and anti-contamination mechanisms.
Timeline (last 90 days)
SWE-Race is an AI programming benchmark targeting concurrency bugs, extracting 188 real concurrency defects from merged PRs across approximately 100 Python projects.
Each task in SWE-Race is scored by the project's own test cases, runs in a container without network connectivity, and the repository is trimmed to a single commit to prevent retrieving fixes from git history.
The team reviewed all approximately 11,000 commands executed by the agent and found that all 69 attempts to access the network failed.
The SWE-Race evaluation protocol follows DeepSWE's 100-step setting, limiting the maximum number of action steps for the agent.
Half of SWE-Race's tasks are private for cross-validation, and the scores of three models on the public set and private set are consistent.
Under the condition of only one attempt per task, GLM-5.3 Flash scored 85%.
About half of the tasks are easy for all models, with scores close to 100%; on the other half of difficult tasks, the three models scored 50%, 45%, and 23% respectively.
The capability gap between models is determined almost entirely by the difficult tasks.
The team compared earlier bugs (before 2026) with newer bugs of similar scale and found that earlier issues were resolved at a probability about 9 percentage points higher, but the confidence interval for this difference spans zero, making it inconclusive.
The SWE-Race leaderboard annotates each score with the number of attempts and confidence intervals.
1 more timeline events
All Facts (11)
SWE-Race is an AI programming benchmark targeting concurrency bugs, extracting 188 real concurrency defects from merged PRs across approximately 100 Python projects.
50%UnverifiedEach task in SWE-Race is scored by the project's own test cases, runs in a container without network connectivity, and the repository is trimmed to a single commit to prevent retrieving fixes from git history.
50%UnverifiedThe team reviewed all approximately 11,000 commands executed by the agent and found that all 69 attempts to access the network failed.
50%UnverifiedThe SWE-Race evaluation protocol follows DeepSWE's 100-step setting, limiting the maximum number of action steps for the agent.
50%UnverifiedHalf of SWE-Race's tasks are private for cross-validation, and the scores of three models on the public set and private set are consistent.
50%UnverifiedUnder the condition of only one attempt per task, GLM-5.3 Flash scored 85%.
50%UnverifiedAbout half of the tasks are easy for all models, with scores close to 100%; on the other half of difficult tasks, the three models scored 50%, 45%, and 23% respectively.
50%UnverifiedThe capability gap between models is determined almost entirely by the difficult tasks.
50%UnverifiedThe team compared earlier bugs (before 2026) with newer bugs of similar scale and found that earlier issues were resolved at a probability about 9 percentage points higher, but the confidence interval for this difference spans zero, making it inconclusive.
50%UnverifiedThe SWE-Race leaderboard annotates each score with the number of attempts and confidence intervals.
50%UnverifiedThe SWE-Race task dataset has been open-sourced on Hugging Face (evaligo/swe-race), and run logs can be viewed at labs.evaligo.com/swe-race.
50%