DeepSWE
A benchmark designed specifically for evaluating large models on software engineering tasks. Core scenarios include automatic bug fixing in GitHub repositories, code generation, and multi-file refactoring. It corresponds to the SWE-bench task set and is more reflective of real-world development than general-purpose benchmarks.
Core Facts
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
DeepSWE is a code capability evaluation benchmark for software engineering scenarios, with tasks sourced from real GitHub Issue fix scenarios.
GPT-6.1 Sol performs best on the DeepSWE benchmark at the high reasoning effort level
Timeline (last 90 days)
DeepSWE focuses on complex, deeply dependent engineering problems; SWE-Marathon emphasizes long-chain, multi-stage tasks, simulating scenarios of sustained work over long periods.
Meta's chart shows Muse Wonder beating Claude Opus 5.5 on long-horizon coding tasks: DeepSWE score 78.8 versus 74.2, SWE-Marathon score 53.1 versus 46.3.
Muse Wonder Full holds a leading position in long-horizon coding benchmarks such as DeepSuite.
On the DeepSWE coding leaderboard, Beam scored 44.4 points, ranking last, below DeepSeek V4.1 Flash's 74.2 points, Kimi K3's approximately 68 points, GLM's approximately 61 points, and Qwen3 Max's approximately 51 points.
On the DeepSWE 1.1 coding benchmark, Mistral Large 4 Preview scored 62, ranking second, behind only Kimi K3 paired with Kimi Code CLI at 68.
The SWE-Race evaluation protocol follows DeepSWE's 100-step setting, limiting the maximum number of action steps for the agent.
In both the DeepSWE and Terminal-Bench evaluations, Replit Agent achieved Pareto optimality relative to Astra running alone, with no published Astra baseline achieving both lower cost and higher score.
Argon achieved a score of 77.9% on the DeepSWE benchmark, setting a new record
Google reports that Gemini 4 Argon achieved a score of 77.9% on the DeepSWE 1.1 software engineering benchmark (as self-reported by the vendor)
Argon set a new record of 77.9% on DeepSWE v1.1, which measures real long-horizon software engineering tasks.
30 more timeline events
All Facts (20)
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
85%VerifiedDeepSWE is a code capability evaluation benchmark for software engineering scenarios, with tasks sourced from real GitHub Issue fix scenarios.
75%VerifiedGPT-6.1 Sol performs best on the DeepSWE benchmark at the high reasoning effort level
70%VerifiedV4.1 Flash surpassed Opus 5 on the DeepSWE benchmark by a narrow margin of 0.2 points.
70%VerifiedOn the DeepSWE 1.1 coding benchmark, Mistral Large 4 Preview scored 62, ranking second, behind only Kimi K3 paired with Kimi Code CLI at 68.
65%Partially VerifiedMeta's chart shows Muse Wonder beating Claude Opus 5.5 on long-horizon coding tasks: DeepSWE score 78.8 versus 74.2, SWE-Marathon score 53.1 versus 46.3.
65%UnverifiedDeepSeek officially announced that V4 Pro scored as high as 87.9 on the Terminal Bench benchmark.
60%UnverifiedDeepSWE focuses on complex, deeply dependent engineering problems; SWE-Marathon emphasizes long-chain, multi-stage tasks, simulating scenarios of sustained work over long periods.
50%UnverifiedMuse Wonder Full holds a leading position in long-horizon coding benchmarks such as DeepSuite.
50%UnverifiedOn the DeepSWE coding leaderboard, Beam scored 44.4 points, ranking last, below DeepSeek V4.1 Flash's 74.2 points, Kimi K3's approximately 68 points, GLM's approximately 61 points, and Qwen3 Max's approximately 51 points.
50%UnverifiedThe SWE-Race evaluation protocol follows DeepSWE's 100-step setting, limiting the maximum number of action steps for the agent.
50%UnverifiedIn both the DeepSWE and Terminal-Bench evaluations, Replit Agent achieved Pareto optimality relative to Astra running alone, with no published Astra baseline achieving both lower cost and higher score.
50%UnverifiedArgon achieved a score of 77.9% on the DeepSWE benchmark, setting a new record
50%UnverifiedGoogle reports that Gemini 4 Argon achieved a score of 77.9% on the DeepSWE 1.1 software engineering benchmark (as self-reported by the vendor)
50%UnverifiedArgon set a new record of 77.9% on DeepSWE v1.1, which measures real long-horizon software engineering tasks.
50%UnverifiedIn the DeepSWE 1.1 benchmark, GPT-6.1 Sol scored 75.2%, Astra scored 74.8%, Cloud Sonnet 5.5 scored 71.0%, and GPT-6 Sol scored a top of 68.8%.
50%UnverifiedAfter lightweight fine-tuning, CLM-8B set a new SOTA of 81.6% (Pass@1) on agentic coding benchmarks such as DeepSWE.
50%UnverifiedHandshake's audit of DeepSWE found that frontier coding agents reason about the hidden scorer in more than 80% of sampled execution traces.
50%UnverifiedAccording to official statements, V4.1 Flash surpasses GPT-5.6 SOAR and Opus 5 on multiple benchmarks including Terminal Bench 2.1, DeepSWE 1.1, CyberGym, HLE, Automation Bench, and Agent's Last Exam.
50%UnverifiedAfter fine-tuning on agent trajectories, CLM achieved 87.6% on Terminal-Bench 2.1 and 81.6% on DeepSWE.
50%