SWE-CoWork
SWE-CoWork is a benchmark for evaluating the ability of coding agents to complete end-to-end tasks in a simulated software engineering organization. It contains 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains, emphasizing collaborative aspects such as communication, requirement tracking, and verification.
Timeline (last 90 days)
SWE-CoWork is a benchmark for measuring the capabilities of coding agents as "digital employees", comprising 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains.
Each task in SWE-CoWork simulates an engineering organization, with 26 colleagues around the agent, each having clear responsibilities, limited knowledge boundaries, and deliberately controlled collaborative behavior.
In the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.
The research team evaluated 11 models on SWE-CoWork.
Controlled experiments show that after providing models with complete requirements, the F2P pass rates of Seed Evolving and GPT-6 Astra increased by about 16 percentage points, but the full task success rate did not change at all.
In the requirement-level trajectory analysis, Claude-Opus-5.5 could on average accurately direct 93% of first inquiries to the owner of the queried card, but ultimately obtained only 64% of the requirements held by colleagues.
In the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.
In SWE-CoWork, agent failures run through the entire process of clarification, implementation, validation, and handoff, including deferred questions with no follow-up, incomplete implementation of obtained requirements, and self-perceived completed changes that were not tested.
For teams hoping to deploy coding agents as digital employees, evaluation criteria should focus on whether the agent can follow a task from beginning to end, not merely on the number of bugs fixed.
All Facts (9)
SWE-CoWork is a benchmark for measuring the capabilities of coding agents as "digital employees", comprising 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains.
50%UnverifiedEach task in SWE-CoWork simulates an engineering organization, with 26 colleagues around the agent, each having clear responsibilities, limited knowledge boundaries, and deliberately controlled collaborative behavior.
50%UnverifiedIn the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.
50%UnverifiedThe research team evaluated 11 models on SWE-CoWork.
50%UnverifiedControlled experiments show that after providing models with complete requirements, the F2P pass rates of Seed Evolving and GPT-6 Astra increased by about 16 percentage points, but the full task success rate did not change at all.
50%UnverifiedIn the requirement-level trajectory analysis, Claude-Opus-5.5 could on average accurately direct 93% of first inquiries to the owner of the queried card, but ultimately obtained only 64% of the requirements held by colleagues.
50%UnverifiedIn the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.
50%UnverifiedIn SWE-CoWork, agent failures run through the entire process of clarification, implementation, validation, and handoff, including deferred questions with no follow-up, incomplete implementation of obtained requirements, and self-perceived completed changes that were not tested.
50%UnverifiedFor teams hoping to deploy coding agents as digital employees, evaluation criteria should focus on whether the agent can follow a task from beginning to end, not merely on the number of bugs fixed.
50%