[KongchangAI]
Benchmark

SWE-CoWork

SWE-CoWork is a benchmark for evaluating the ability of coding agents to complete end-to-end tasks in a simulated software engineering organization. It contains 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains, emphasizing collaborative aspects such as communication, requirement tracking, and verification.

Timeline (last 90 days)

Nov 17

SWE-CoWork is a benchmark for measuring the capabilities of coding agents as "digital employees", comprising 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains.

Unverified50%
Nov 17

Each task in SWE-CoWork simulates an engineering organization, with 26 colleagues around the agent, each having clear responsibilities, limited knowledge boundaries, and deliberately controlled collaborative behavior.

Unverified50%
Nov 17

In the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.

Unverified50%
Nov 17

The research team evaluated 11 models on SWE-CoWork.

Unverified50%
Nov 17

Controlled experiments show that after providing models with complete requirements, the F2P pass rates of Seed Evolving and GPT-6 Astra increased by about 16 percentage points, but the full task success rate did not change at all.

Unverified50%
Nov 17

In the requirement-level trajectory analysis, Claude-Opus-5.5 could on average accurately direct 93% of first inquiries to the owner of the queried card, but ultimately obtained only 64% of the requirements held by colleagues.

Unverified50%
Nov 17

In the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.

Unverified50%
Nov 17

In SWE-CoWork, agent failures run through the entire process of clarification, implementation, validation, and handoff, including deferred questions with no follow-up, incomplete implementation of obtained requirements, and self-perceived completed changes that were not tested.

Unverified50%
Nov 17

For teams hoping to deploy coding agents as digital employees, evaluation criteria should focus on whether the agent can follow a task from beginning to end, not merely on the number of bugs fixed.

Unverified50%

All Facts (9)

Unverified

SWE-CoWork is a benchmark for measuring the capabilities of coding agents as "digital employees", comprising 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains.

50%
Unverified

Each task in SWE-CoWork simulates an engineering organization, with 26 colleagues around the agent, each having clear responsibilities, limited knowledge boundaries, and deliberately controlled collaborative behavior.

50%
Unverified

In the SWE-CoWork evaluation, the two best-performing models pass about 63% of the fail-to-pass (F2P) tests, but fully complete fewer than 8% of the tasks.

50%
Unverified

The research team evaluated 11 models on SWE-CoWork.

50%
Unverified

Controlled experiments show that after providing models with complete requirements, the F2P pass rates of Seed Evolving and GPT-6 Astra increased by about 16 percentage points, but the full task success rate did not change at all.

50%
Unverified

In the requirement-level trajectory analysis, Claude-Opus-5.5 could on average accurately direct 93% of first inquiries to the owner of the queried card, but ultimately obtained only 64% of the requirements held by colleagues.

50%
Unverified

In the SWE-CoWork trajectory analysis, 31% of the requirements that models reported as completed could not pass the tests.

50%
Unverified

In SWE-CoWork, agent failures run through the entire process of clarification, implementation, validation, and handoff, including deferred questions with no follow-up, incomplete implementation of obtained requirements, and self-perceived completed changes that were not tested.

50%
Unverified

For teams hoping to deploy coding agents as digital employees, evaluation criteria should focus on whether the agent can follow a task from beginning to end, not merely on the number of bugs fixed.

50%

Source Articles