SWE-CoWork: Why Do Coding Agents Struggle as "Digital Employees"?

SWE-CoWork benchmark reveals: coding agents can fix code correctly but almost never complete full tasks.
SWE-CoWork is a coding agent benchmark simulating real software organizations: 101 tasks, 101 real repos, 11 languages, 12 domains, with agents collaborating with 26 colleagues of limited responsibilities to clarify requirements. Evaluating 11 models found top performers pass ~63% of fail-to-pass tests but complete under 8% of full tasks. Even with complete requirements provided, F2P rates rise ~16 points while full success stays unchanged—the bottleneck is post-acquisition processing. Trajectory analysis shows failures span clarification, implementation, validation, and handoff; continuous requirement tracking and state maintenance are key.
In real software organizations, coding agents face far more than just writing code. They need to figure out what to do, find the right person to clarify requirements, make judgments when information conflicts, and finally ensure the deliverable actually passes tests. SWE-CoWork is a benchmark designed to measure this kind of "digital employee" capability, and its results reveal an awkward reality: models can write most of the code correctly, yet almost never complete a full task.

A testing ground closer to the real workplace
SWE-CoWork contains 101 development tasks spanning 101 real code repositories, 11 programming languages, and 12 application domains. Unlike typical single-turn coding evaluations, each task places the agent inside a simulated engineering organization: surrounded by 26 colleagues, each with clearly defined responsibilities, limited knowledge boundaries, and deliberately controlled collaborative behavior.
The agent receives only a brief handoff note; the real requirements are scattered across colleagues and work records. It must proactively ask questions, cross-check, handle contradictory evidence, and finally submit changes, with success judged by private executable tests. This setup makes "communication" and "requirement tracking" mandatory assessment criteria rather than optional bonuses.
A stark gap: code gets written, tasks don't get delivered
The research team evaluated 11 models, and the core finding is a huge gap between partial implementation and full delivery. The two best-performing models pass about 63% of the fail-to-pass (F2P) tests, yet complete fewer than 8% of full tasks.
This contrast is telling: models are already quite skilled at "fixing a piece of logic," but once required to close the loop on a task from start to finish, success rates plummet. F2P tests measure whether a fix takes effect, whereas full task success also requires that requirement understanding, scoping, and validation all hold up.
fail-to-pass (F2P) tests are a common category of test cases in software engineering evaluation: they fail before the fix (fail) and should pass after a correct modification (pass). They directly verify whether a code change truly resolves the target defect, regardless of the state of other existing tests in the repository. Compared with end-to-end acceptance, F2P tests cover only the local signal of "whether the fix takes effect," and are therefore often used as a fine-grained metric of code modification correctness. But their limitations are obvious too: passing F2P does not mean the requirement is fully implemented, nor does it mean there are no problems with handoff and validation in the collaborative workflow. SWE-CoWork leverages the gap between F2P and full task success to expose the capability disconnect where models "can modify code but cannot deliver tasks."
If the requirements are fully provided, why still incomplete?
A seemingly reasonable hypothesis is that models fail because they don't gather all the requirements. So the researchers ran a controlled experiment, providing the complete requirements directly to the model. The results showed that Seed Evolving and GPT-6 Astra improved F2P pass rates by about 16 percentage points, but full task success rates did not change at all.
This evidence is significant. It means the bottleneck is not only at the information-gathering end, but also in processing and executing once the information is in hand. Even with requirements in front of them, models can still fail in implementation, validation, or handoff.
Tracing trajectories: failures happen at every step
Analysis of requirement-level trajectories further breaks down where failures occur. Across multiple runs, Claude-Opus-5.5 could, on average, accurately direct 93% of its first questions to the owner of the card being queried—quite good at "finding the right person." But it ultimately obtained only 64% of the requirements held by colleagues.
More interesting are several systematic problems: deferred questions had no follow-up; even when requirements were obtained, implementation remained incomplete; and 31% of requirements reported as completed failed to pass tests. In other words, failures are not concentrated in one stage but run through the entire process of clarification, implementation, validation, and handoff.
Requirement-level trajectories refer to the complete behavioral sequence formed by an agent interacting with colleagues, obtaining and confirming requirements during a task, including whom it asks, what it asks, whether it follows up on deferred questions, and when it considers requirements obtained. Unlike evaluations that only examine the final code diff, trajectory analysis treats the agent's communication and decision-making process itself as the object of evaluation, making it possible to pinpoint whether failures occur in clarification, implementation, validation, or handoff. This analysis relies on structured records of each round of dialogue and tool calls, and can distinguish between two fundamentally different types of failure—"didn't ask" versus "asked but didn't use"—thereby providing actionable diagnostic evidence for improving agents' state-maintenance capabilities.
Implications for autonomous programming
The value of SWE-CoWork lies in restoring "software engineering" to what it really is—a collaborative activity that depends on continuous communication and requirement tracking. Current models perform impressively on isolated capabilities but lack the state-maintenance ability to carry a task through from beginning to end: questions asked must be remembered and followed up, obtained requirements must be translated into code, and written code must be verified by the model itself.
For teams hoping to deploy coding agents as digital employees, this benchmark suggests a pragmatic criterion: don't just look at how many bugs it can fix; look at whether it can follow a task from start to finish. Continuous requirement tracking—from clarification to implementation, validation, and handoff—is the key challenge of reliable autonomous software engineering.
State-maintenance capability is an autonomous agent's ability to maintain a consistent understanding of goals, known information, and pending items over long-horizon tasks. In multi-turn collaborative scenarios like SWE-CoWork, the agent needs to remember which questions remain unanswered, which requirements have been confirmed, and which changes still need validation, and proactively revisit these states in subsequent steps. A lack of state maintenance leads to typical problems: deferred questions with no follow-up, obtained requirements not translated into implementation, and changes believed complete delivered without testing. This is fundamentally different from the capability required for single-turn code generation—the latter only requires producing a correct snippet within a limited context, while the former requires continuously aligning goals and evidence across a timeline spanning multiple interactions.
One-line Summary
SWE-CoWork benchmark reveals: coding agents can fix code correctly, yet almost never complete full tasks.
Paragraph Summary
SWE-CoWork is a benchmark for coding agents that simulates real software organizations, with 101 tasks, 101 real repositories, 11 languages, and 12 domains; in each task the agent must collaborate with 26 colleagues with limited responsibilities to clarify requirements. Evaluating 11 models found that the best performers pass about 63% of fail-to-pass tests but complete fewer than 8% of full tasks. A controlled experiment shows that even when complete requirements are provided directly, F2P pass rates rise by about 16 percentage points while full task success rates remain unchanged, indicating the bottleneck lies in processing and executing information once obtained. Trajectory analysis further shows failures run through clarification, implementation, validation, and handoff, and that continuous requirement tracking and state maintenance are the key to reliable autonomous programming.
Related articles

FBI Warns About AI Hacking Tool Artex: Systems Breached in Three Hours
FBI warns about AI hacking tool Artex: describe targets in natural language for automated scanning and intrusion, with some attacks achieving full privileged access within three hours. It affected over 2,000 Microsoft 365 organizations and South Korean banks. Copies still circulate after the developer took it down in October; security fundamentals remain key.

FBI-Warned AI Hacking Tool ARTEX: Cost Disruption and Defensive Restructuring
The FBI-warned AI hacking tool ARTEX reduces penetration testing costs to $150-$1,500, causing financial data breaches and supply chain attacks. Learn how it works, real cases, and a five-step defense strategy.

FBI Warns: ARTEX AI Hacking Tool Breaches Systems in Three Hours
FBI warns the open-source AI pentesting framework ARTEX is being abused, with some intrusions gaining privileged access in under three hours, over 1,600 consoles exposed, and links to South Korean financial data breaches and the ConnectWise supply chain attack.