T3MP3ST: The Open-Source Framework That Turns Coding Agents into Autonomous Red Team Operators

When Your Coding Assistant Grows Fangs

You might already have Claude Code, Codex, or Hermes running in your terminal, writing code and squashing bugs. Now an open-source project called T3MP3ST wants to strap a full offensive security exoskeleton onto these agents — transforming them into autonomous red team units capable of hunting real-world vulnerabilities.

Offensive Security & Red Teams — Background: Offensive security is the practice of proactively simulating attacker behavior to discover and remediate vulnerabilities, as opposed to defensive security. The concept of a red team originates from Cold War military exercises, referring to a team that plays the attacker role — distinct from the defending "blue team" and the hybrid "purple team." In commercial contexts, a full red team engagement can cost hundreds of thousands of dollars and require weeks of effort from elite security professionals. That's exactly the barrier T3MP3ST claims to lower.

In the author's own words, this is a "harness of harnesses" — it doesn't build a new model, but layers offensive cybersecurity prompt engineering and a complete exploitation toolchain on top of coding agents you're already paying for. Point it at an authorized target, click a few times, and watch it autonomously start hunting vulns.

The project is fully open-source and free under the AGPL-3.0 license. Its core thesis: "Offensive security shouldn't be a pay-to-play game."

Why AGPL-3.0 Matters: AGPL (Affero General Public License v3) is a variant of the GNU GPL designed specifically for networked services. The key difference from standard GPL: even if you only provide the software as a network service without distributing it, you must still open-source your modifications. This is significant for the security tool ecosystem — commercial security companies can't quietly wrap it into a paid SaaS offering without giving back to the community, effectively preventing "open-source washing."

Coverage: From Web Apps to SCADA

T3MP3ST's claimed capabilities span nearly every major red team battleground:

• Web Applications & APIs: Classic vulnerabilities from the OWASP Top 10
• Network Reconnaissance: Real-time fingerprinting via nmap / DNS / HTTP; lateral movement and privilege escalation are still experimental
• Source Code Auditing: White-box vulnerability discovery
• CTF & Lab Environments: Wargames and challenge repositories
• Smart Contracts / DeFi / Solidity: Note — limited to reproduction (e.g., Damn Vulnerable DeFi), not novel vulnerability discovery
• Embedded Systems, IoT, OT/SCADA, and Robotic Open-Source Systems

What Is the OWASP Top 10?: OWASP (Open Web Application Security Project) is the world's most influential web security nonprofit. Its Top 10 list is the most widely cited standard for web application security risks, updated every few years. The 2021 edition covers injection attacks (SQL/command injection), broken authentication, sensitive data exposure, broken access control, security misconfigurations, XSS, and insecure deserialization, among others. Over 90% of web security training programs and penetration test reports use it as a reference framework.

This list speaks to the author's ambition — not a point solution, but full coverage of the Cyber Kill Chain.

The Cyber Kill Chain: Proposed by Lockheed Martin in 2011, this model breaks a complete cyberattack into seven sequential stages: Reconnaissance → Weaponization → Delivery → Exploitation → Installation → Command & Control (C2) → Actions on Objectives. The key insight: defenders only need to disrupt one stage to break the chain, while attackers must succeed at every step.

The Numbers: Three Benchmark Suites

What really makes T3MP3ST worth watching is its reproducible evaluation data. The author explicitly cross-checks every captured flag against the challenge's own flag oracle (reported vs. expected), and uses verify-claims to recalculate pass rates from submitted artifacts — a deliberate defense against "benchmark gaming" accusations.

XBEN: 90.1% Black-Box, 98.7% White-Box

On XBOW's own 104-challenge benchmark suite XBEN (using gpt-5.5):

• Black-box: Single-agent exploit loops achieved 90.1% pass@1, with the worst single run at 87.5% (91/104) — surpassing XBOW's own self-reported 85%.
• White-box (source provided in stages, reported separately): pass@1 reached 98.7%, worst single run at 98.1% (102/104).

Understanding pass@k: pass@k is the standard evaluation metric in code generation and problem-solving. pass@1 means the model solves the problem on a single attempt — the strictest measure, since it offers no "try again until you get it" allowance. pass@10 allows ten attempts and naturally yields higher success rates. In security testing, pass@1 matters most: real penetration tests rarely offer a second chance, since a single triggered alert can blow the entire operation.

The author quipped: "Looks like we need new benchmarks."

Cybench: Real Exploitation on an Academic Benchmark

On the 40-task academic benchmark Cybench (using Opus 4.8, with hints and solutions removed), T3MP3ST achieved 23/40 = 58% single-attempt, no-hint pass@1. These aren't toy problems — they include format string pwn challenges, eval sandbox escapes, and cryptographic oracle attacks. For reference, Anthropic's official reported figure is 76.5% pass@10.

Why Format Strings and Sandbox Escapes Are Hard: These are among the most demanding scenarios in both CTF competitions and real-world penetration testing. Format string vulnerabilities require precise memory layout control, often involving dozens of carefully orchestrated steps in 64-bit programs. Sandbox escape requires finding boundary flaws in restricted execution environments — a core challenge in cloud and browser security. These problems are difficult to automate with AI precisely because they demand deep contextual reasoning and exact multi-step operation chains. That's what makes T3MP3ST's 58% pass@1 noteworthy.

CVE-Zero: Real Vulnerabilities After Training Cutoff

The most compelling test is CVE-Zero. The author cold-started the system (COLD) and pointed it at real CVEs disclosed after the model's training cutoff:

• 10 CVEs the model had never seen, spanning 7 programming languages, with no prompt tuning targeting them
• A single agent pinpointed 8/10 to the exact file, line number, and CWE classification
• Full pack mode surfaced all 10

CVE and CWE Systems: CVE (Common Vulnerabilities and Exposures) is MITRE's globally unified vulnerability naming system, assigning unique identifiers to known vulnerabilities (e.g., CVE-2024-12345). The companion CWE (Common Weakness Enumeration) classifies vulnerability types — CWE-79 is XSS, CWE-89 is SQL injection. The critical design of CVE-Zero: using CVEs disclosed after the training cutoff means the model cannot "cheat" by recalling memorized training data. It must rely on genuine code reasoning ability. This is a rigorous "data contamination exclusion" method in AI security evaluation.

The elegance of this design: it simultaneously eliminates memorization (the vulnerability isn't in the training data) and overfitting (the prompts were never tuned for these cases). The author honestly notes the small sample size (n=10) and calls the results "directional."

Architecture: Solo Operator and Swarm Hunting

T3MP3ST offers two operating modes — the most architecturally interesting aspect of the project.

Single-Agent Mode: The path used in all benchmark tests above. Already quite capable on its own.

Pack-Hunt (Swarm) Mode: Dozens of agents distributed across 8 specialist operator roles, mapped to stages of the Cyber Kill Chain and MITRE ATT&CK: recon → scan → exploit → lateral → exfil → persistence → C2 → report.

MITRE ATT&CK Framework: ATT&CK (Adversarial Tactics, Techniques & Common Knowledge) is a more granular adversarial tactics knowledge base than the Cyber Kill Chain, continuously maintained by MITRE. It breaks attack behaviors into 14 tactic categories and hundreds of specific techniques, each documented with real-world APT (Advanced Persistent Threat) group usage examples. ATT&CK has become the shared language of red/blue team operations: defenders use it to identify coverage gaps, attackers use it to design attack paths, and security products use it to map detection capabilities. By aligning agent roles to ATT&CK stages, T3MP3ST's swarm output can be directly mapped to industry-standard risk assessment frameworks.

A role called the Op Admiral plans the entire operation from a plain-English objective. With coordination enabled (experimental), operators share a blackboard: a tool-verified finding triggers the next action. The Admiral can also dynamically update other agents' prompts, tools, and configurations mid-operation. The author emphasizes that T3MP3ST "gets stronger the more you use it, as it accumulates memory."

Blackboard Architecture: A classic multi-agent coordination pattern from AI research, originating in the 1970s HEARSAY speech recognition project. The core idea: multiple specialized knowledge sources share a central data structure (the "blackboard"), each independently reading and writing, and deciding when to activate based on changes they observe — like a group of experts collaborating around a shared whiteboard. The advantage is loose coupling: each agent only needs to understand its own domain, not the internals of other agents. In a security context, a recon agent writes discovered ports to the blackboard, the vulnerability scanning agent automatically responds and gets to work, creating a self-driving attack pipeline.

The Arsenal: From nmap to Metasploit

T3MP3ST's "Arsenal" integrates a wide range of mature penetration testing tools: nmap, nuclei, semgrep, ffuf, gobuster, and more. The default configuration includes 35 tools (clean benchmark runs use only bash to ensure comparability); enabling T3MP3ST_FULL_ARSENAL expands this to 83.

Core Pentest Tools at a Glance: nmap is the Swiss Army knife of network discovery and security auditing, capable of port scanning, service version detection, and OS fingerprinting. nuclei is a template-based fast vulnerability scanner with thousands of community-maintained detection templates. semgrep is a source code static analysis tool that detects vulnerability patterns across languages using custom rules. ffuf and gobuster are directory and file brute-force enumeration tools for discovering hidden web resources. All are open-source and standard fixtures in any modern penetration tester's toolkit.

Notable in the security boundary design: "spicy" post-exploitation tools like Metasploit and Hydra are gated behind human approval. Tools are exposed via CLI and HTTP API, and reconnaissance functionality (security_recon) also supports the MCP protocol, enabling native invocation from your agent.

MCP: The New Standard for AI Tool Calling: MCP (Model Context Protocol) is an open standard published by Anthropic in late 2024, designed to unify how AI models interact with external tools and data sources. Before MCP, every AI application needed custom integration code for each tool, creating a fragmented "N×M" integration problem. MCP defines a standardized server-client architecture so any MCP-compatible AI assistant can discover and call tool services in a plug-and-play fashion. By exposing its reconnaissance functionality as an MCP server, T3MP3ST lets users invoke its security scanning capabilities directly from Claude Desktop, Cursor, or any MCP-compatible client — no bespoke integration code required.

A Sober Assessment: This Is v1, Risks Are Real

Despite the impressive numbers, the author remains notably measured. This is a v1 release — portions of the arsenal, swarm coordination, and some lab environments are still being wired up. The project is being built in the open, and "receipts" clearly indicate what's live versus what's on the roadmap. The stated future direction is a self-improving swarm of specialist operators with a Kali+ arsenal and built-in train/test set splits to prevent self-deception on internal evaluations.

Legal and ethical risks must be taken seriously. The author repeatedly emphasizes: authorized use only — point it exclusively at systems you own or have explicit written permission to test. Unauthorized access may constitute a criminal offense, and responsibility lies entirely with the user. The project is delivered under AGPL-3.0 "as-is," with no warranty, no liability, and zero endorsement.

The Legal Boundaries of Unauthorized Access: In virtually every major jurisdiction, unauthorized access to computer systems is a criminal offense. The U.S. Computer Fraud and Abuse Act (CFAA), the EU's Budapest Convention on Cybercrime, and China's Cybersecurity Law all have clear provisions on this. Even well-intentioned security research can carry legal risk without written authorization — there are historical cases of researchers being prosecuted despite acting in good faith. Professional penetration testing requires a complete chain of authorization documents specifying scope, time windows, and permitted techniques. Bug bounty platforms (like HackerOne and Bugcrowd) offer a standardized authorization framework and are the recommended path for legal security research.

Closing Thoughts: The Double-Edged Sword of Security Democratization

T3MP3ST represents an accelerating trend: general-purpose coding agents being repurposed into highly specialized, autonomous security tools. It places red team capabilities in the hands of anyone who has a coding agent — which is both a step forward in democratizing security and a double-edged sword with serious potential for misuse.

For defenders, the dramatically lowered barrier to entry means attack surface exposure will be discovered faster and more cheaply — which in turn forces defensive systems to evolve at the same pace. The long-standing asymmetry in security — attackers only need to find one vulnerability while defenders must plug them all — becomes even sharper when AI-powered automation enters the equation.

The technology is loaded. How you use it is entirely your own choice.