Deep SWE
A software engineering agent benchmark released by Artificial Analysis, requiring models to autonomously complete code repair tasks in real GitHub repositories. It focuses on evaluating end-to-end agent execution capabilities.
Core Facts
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
In a chart published by Google, Argon's Deep SWE score is close to 78%, while Claude Opus 5.5 and GPT-6 Astra are only slightly above 74%.
Timeline (last 90 days)
The Deep SWE score rose from 74.2 to 77, an improvement of 2.8 points.
Gemini 4 Argon achieved a SOTA score of 77.9% on the DeepSWE v1.1 benchmark.
Google states that Argon has set a new state-of-the-art (SOTA) on Deep SWE.
In a chart published by Google, Argon's Deep SWE score is close to 78%, while Claude Opus 5.5 and GPT-6 Astra are only slightly above 74%.
In the Deep SWE benchmark, GPT-6.1 Sol's score at the Low tier is comparable to Astra's at the High tier, but the Low tier costs only 21 cents, while Astra requires $1.46.
The Deep SWE test simulates a real software engineering environment, requiring AI to understand architecture, locate problems, iteratively modify code, and verify results through automated tests within an existing codebase.
In the DeepSWE benchmark, the cost of a single low-tier run for GPT-4.1 Sol is approximately $0.21, while Astra at the same tier requires $1.46, and Opus 5.5 at the max tier needs $14.65 to achieve the same score.
DeepSeek V4.1 Flash scored 74 on the Deep SWE software engineering code test, making it the first open-weight model the UP host has seen to reach 74 on Deep SWE.
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
Compared to the previous generation Qwen3.6-27B, Qwen3.8-27B shows significant improvements on multiple benchmarks, with Deep SWE rising from 13.3 to 42.2 (nearly tripling) and OS World rising from 63.9 to 84.3.
All Facts (8)
The DeepSWE series of benchmarks is used to measure a model's autonomous end-to-end engineering capabilities in real-world software engineering scenarios.
85%VerifiedIn a chart published by Google, Argon's Deep SWE score is close to 78%, while Claude Opus 5.5 and GPT-6 Astra are only slightly above 74%.
65%UnverifiedThe Deep SWE score rose from 74.2 to 77, an improvement of 2.8 points.
50%UnverifiedGemini 4 Argon achieved a SOTA score of 77.9% on the DeepSWE v1.1 benchmark.
50%UnverifiedIn the Deep SWE benchmark, GPT-6.1 Sol's score at the Low tier is comparable to Astra's at the High tier, but the Low tier costs only 21 cents, while Astra requires $1.46.
50%UnverifiedThe Deep SWE test simulates a real software engineering environment, requiring AI to understand architecture, locate problems, iteratively modify code, and verify results through automated tests within an existing codebase.
50%UnverifiedDeepSeek V4.1 Flash scored 74 on the Deep SWE software engineering code test, making it the first open-weight model the UP host has seen to reach 74 on Deep SWE.
50%UnverifiedCompared to the previous generation Qwen3.6-27B, Qwen3.8-27B shows significant improvements on multiple benchmarks, with Deep SWE rising from 13.3 to 42.2 (nearly tripling) and OS World rising from 63.9 to 84.3.
50%