Open-Source Offense-Defense Capabilities Approach Closed-Source, But Coding Agents Are Still Far from "Fully Automated"

In the November 9 AI morning briefing, three evaluation results painted an intriguing picture: open-source models have already approached the closed-source ceiling in high-risk scenarios like cyber offense-defense, yet general-purpose coding agents can't even complete three out of ten tasks fully, while improvements to memory mechanisms are teaching agents to "review before bed."

Cyber Offense-Defense: Open-Source Models Approach the Frontier at One-Tenth the Cost

Evaluation organization Wouse AI released its cyber offense-defense benchmark Cyber Offense, in which AI agents attack two simulated enterprise networks without human assistance. The freely downloadable DeepSig V4.1 Flash scored 72.8%, second only to OpenAI's restricted GPT-6Soft (80.8%). On one of the networks, DeepSig tied with its rival.

More noteworthy is the cost: DeepSig runs at under $3 per attempt, roughly one-tenth of its competitor's. This means open-weight models have scalable economics in scenarios like security offense-defense that require repeated trial and error.

Wouse AI also cautioned that once weights are public, no one can monitor how they're used. This risk isn't theoretical—open-sourcing offense-defense capabilities is tantamount to handing a reusable intrusion methodology to everyone.

The corresponding recommendation is that defenders should seize the window to test and patch their own systems with these models first. The offense-defense asymmetry is further amplified in the AI era: attackers can use low-cost models to probe at scale, while defenders must complete self-checks before vulnerabilities are exploited.

Open-weight models refer to AI models whose parameters (weights) are publicly released as downloadable files, allowing anyone to deploy locally, fine-tune, or even redistribute them. Typical examples include Llama, Mistral, and the DeepSig series mentioned in this article. In contrast, closed-source models can only be called remotely via API, with weights kept private. The core difference isn't just price: open-source models can run offline, be customized for private use, and aren't subject to service providers' call frequency or content policies, making them preferable in scenarios requiring repeated trial and error or handling sensitive data. The trade-off is that safety guardrails can be removed and abuse is hard to trace—which is the institutional reason the article emphasizes that "once weights are public, no one can monitor them."

Coding Agents: Meeting Nearly 70% of Acceptance Criteria on Average, But Only 26.4% Fully Correct

Researchers from Alibaba and other institutions released the coding benchmark Cody 3 Dench, with 273 tasks drawn from over 26,000 real users' coding needs. The top-ranked GPT-6Extra met nearly 70% of acceptance criteria on average, but only 26.4% of tasks were fully correct.

In other words, nearly three-quarters of results still need humans to finish them. This figure reflects the actual experience better than "70% on average"—between partial passing and fully usable lies the tedious but unavoidable work of debugging, edge cases, and integration adaptation.

The paper also notes that these tasks were deliberately selected as difficult, so 26.4% shouldn't be simply read as "coding agents are failing." A more reasonable interpretation: on the difficult distribution formed by real user needs, current agents excel at accelerating the first half, while reliability in the last quarter still depends on human engineers.

Acceptance criteria are explicit conditions in software engineering that determine whether a task is complete, usually written by the requester before development—for example, "the API returns the specified error code after a timeout" or "response time under 200ms with a thousand concurrent requests." In the benchmark, each task is broken into several automatically verifiable criteria, judged pass or fail individually, so both "average pass rate" and "all-pass rate" can be computed. The latter is stricter because any single edge case failing counts as a failure. This also explains why the gap between nearly 70% on average and only 26.4% fully correct is so large: real requirements often contain many trivial but simultaneously necessary constraints, and agents excel at covering the main logic but tend to stumble on a few edge items.

Memory Mechanisms: Agents Learn to "Dream" and Review

Anthropic published memory research on the Claude agent. Claude "dreams" when idle, reviewing past drawing records, correcting notes, merging duplicates, and even building its own knowledge Wiki.

Anthropic tested on 350,000 internal files. After increasing dreaming from two rounds to 128, the score on answering questions using only the Wiki rose from 13% to 48%, and with search tools, the total score reached 61%.

The leap from 13% to 48% suggests that how memory is organized may be more critical than memory capacity. Letting the model organize its own experience offline is like turning "retrieval" into "structured understanding," which is closer to how humans learn than simply expanding the context window—filing away scattered information before bed makes it faster to recall the next day.

Context window refers to the upper limit on text length a model can read simultaneously in a single inference. Expanding the window is often treated as a way to improve "memory," but it essentially stuffs more raw material into the prompt, and the model still needs to re-retrieve and understand it on each call—cost rises linearly with window size, and information in long contexts is easily diluted. Retrieval-Augmented Generation (RAG) instead retrieves relevant snippets from an external knowledge base and inserts them into the prompt when needed. Claude's "dreaming" mechanism takes a third path: offline, it compresses, deduplicates, and rewrites scattered experience into structured entries (such as a self-built Wiki), so that retrieval targets organized conclusions rather than raw records. This reduces the burden of online inference and explains why the gains from more organization rounds far exceed simply expanding capacity.

Apple's Catch-Up: Siri May Move Toward an Agent Form

According to Bloomberg, Apple's new CEO Zhang Tainas is pushing to accelerate AI progress. Apple is considering making Siri more like an agent as early as next year, able to track ongoing goals and proactively complete follow-up actions.

However, Apple hasn't officially confirmed this, and the features haven't been released. Transforming Siri from a "Q&A entry point" into an agent that "tracks goals continuously and executes proactively" involves system permissions, privacy boundaries, and multi-step task orchestration—exactly where Apple has historically been most cautious. The report's credibility should be discounted, but the industry direction it reflects is clear: voice assistants are being redefined as task agents.

The Common Signal of Three Evaluations

Looking at Cyber Offense, Cody 3 Dench, and the Claude memory research together points to the same conclusion: the capability boundary of AI agents is shifting from "single-turn performance" to "multi-turn reliability and cost efficiency." In offense-defense scenarios, open-source models catch up to closed-source by running repeatedly at low cost; in coding scenarios, single-turn scores look good but full delivery rates are low; in memory scenarios, the gains from offline organization far exceed simply piling on compute.

The direct implication for users: when evaluating agents, don't just look at average scores—look at "all-correct rate" and "cost per run." The former determines whether you need to hire people to finish the job; the latter determines whether this can scale.

One-line Summary

Three evaluations show agent capability boundaries are shifting from single-turn performance to multi-turn reliability and cost efficiency.

Paragraph Summary

The article connects three evaluations from the November 9 AI morning briefing: Wouse AI's Cyber Offense shows open-source DeepSig V4.1 Flash approaches GPT-6Soft's 80.8% with 72.8%, at under $3 per run—about one-tenth the competitor's cost—though public weights amplify abuse risk; in Alibaba and other institutions' Cody 3 Dench coding benchmark, GPT-6Extra meets nearly 70% of acceptance criteria on average but only 26.4% fully correct, indicating agents excel at accelerating the first half while the latter stages still need humans; Anthropic's Claude memory research shows that after adding "dreaming" organization rounds, answering with only the Wiki rose from 13% to 48%, reaching 61% with search. The article also mentions rumors that Apple may move Siri toward an agent form. The conclusion: evaluating agents should focus on all-correct rate and cost per run.