Building an Automated Penetration Testing System with Claude: A Deep Dive into Memory, Skills, and Agent Architecture

A security researcher recently shared an in-depth hands-on video on Bilibili, demonstrating how to customize Claude into a fully automated penetration testing system equipped with memory, skills, and a knowledge base. In the video, the author successfully compromised multiple CTF targets live on camera and showcased a real vulnerability discovered that same day in an actual Bug Bounty program with Claude's assistance. This methodology offers valuable insights into how AI Agents can be practically applied in professional security research.

Bug Bounty programs are initiatives where companies invite external security researchers to discover and report vulnerabilities within an authorized scope, rewarding them with monetary compensation. Major platforms include HackerOne, Bugcrowd, and Intigriti, with participating companies spanning tech giants like Google, Meta, and Microsoft. Each program defines a clear Scope (target boundaries), with vulnerabilities classified by severity and bounties ranging from a few hundred to hundreds of thousands of dollars. Unlike CTF lab environments, Bug Bounty targets are real production systems — this "live-fire training" dynamic makes it the ideal proving ground for evaluating AI-assisted security research.

Core Architecture: Three Key Concepts — Memory, Skills, and Agent

The author breaks the entire system down into three core concepts, emphasizing that these design principles aren't limited to Claude — they apply equally to other mainstream AI coding assistants.

Memory is the foundation of the system. It refers to what the AI must retain across every session — for example: vulnerability submissions must include a PoC (Proof of Concept), clickjacking-type issues don't need to be reported, which documents to consult when hitting a dead end, and whether the AI has permission to spin up sub-Agents. These persistent rules ensure the AI strictly follows the researcher's established workflow in every conversation.

PoC (Proof of Concept) is a core deliverable in security research — the minimal demonstration code or step-by-step procedure that proves a vulnerability is genuinely exploitable. Bug Bounty platforms typically require researchers to submit a complete PoC before a vulnerability can be rated and rewarded. A valid PoC must demonstrate that the attack path is reproducible and the impact is clearly scoped, without causing material damage to the target system. Embedding "must include PoC" into the AI's Memory rules is essentially encoding industry standards as behavioral constraints for the AI, preventing it from outputting theoretical suggestions that can't be acted upon.

Skills are analogous to functions in programming — specific tasks that need to be executed repeatedly. The author gives the example of a dedicated skill for "JavaScript file auditing" that may be invoked thousands of times, which is why he wrote detailed operating instructions for it. He has also built a range of specialized skill modules including reconnaissance (recon), extended recon, WAF bypass, and more.

WAF (Web Application Firewall) is a security appliance deployed in front of web services that blocks malicious requests such as SQL injection and XSS through rule matching. WAF bypass is a high-frequency topic in penetration testing, with common techniques including: encoding transformations (URL encoding, Unicode encoding), case confusion, comment insertion, Chunked Encoding, and exploiting blind spots in WAF regex rules. The advantage of encapsulating WAF bypass as a standalone Skill module is the ability to continuously accumulate target-specific Payload libraries for particular WAF products (e.g., Cloudflare, Akamai, AWS WAF), building reusable institutional knowledge over time.

Agent handles task delegation and Token management. The author configured Agents with different roles: some specialize in "deep analysis" and consume large amounts of Tokens to dissect packets line by line; others handle "Orchestration," automatically determining which tasks to dispatch in which scenarios.

This multi-Agent collaborative design reflects careful Token economics. In real-world LLM deployments, Tokens are both a unit of computation and a unit of cost — complex task context windows can consume tens or even hundreds of thousands of Tokens. By splitting tasks into sub-tasks with different "precision requirements" — activating the high-cost deep-analysis Agent only when necessary, while a low-cost orchestration Agent handles routine scheduling decisions — this closely mirrors the microservices architecture philosophy in software engineering. For security research, which demands long, iterative workflows, thoughtful Agent division can multiply the effective analysis volume achievable within the same budget several times over.

A Key Insight: AI Is a Dynamic Processor, Not a Calculator

The author repeatedly hammers home a core point: don't think of AI as a calculator. A calculator only has predefined operations with all logic hardcoded; LLMs like Claude are "dynamic, context-aware" processors.

He draws an analogy: treat Claude like "a smart but inexperienced worker you need to train." Hand it a massive JavaScript file and ask it to extract data flows, list POST requests, and map out business logic — a task that might take a human five to ten hours can be completed by the AI in five to ten minutes. The fundamental reason for this capability gap is that it can perform dynamic, context-sensitive analysis — something traditional automation tools simply cannot do.

Regarding the popular online analogy of "calculators didn't eliminate math, so AI won't eliminate certain professions either," the author argues the comparison is flawed from the outset: equating a single-operation tool with artificial intelligence puts them in entirely different leagues. AI is a fundamentally new kind of entity with no direct historical parallel.

Hands-On Demo: From CTF Labs to Real Vulnerabilities

In the video, the author tackles multiple targets of increasing difficulty. The first involves a relatively straightforward WAF bypass scenario; the second is an Email Header Injection target requiring deep reasoning.

Email Header Injection is an injection vulnerability in server-side email sending logic. When a web application concatenates user input (such as name or email fields) directly into email headers (e.g., To, CC, Subject) without sanitization, an attacker can insert CRLF (\r\n) characters to append additional headers — redirecting recipients, adding BCC fields for email hijacking, or even injecting MIME boundaries to alter the email body. What makes this vulnerability class distinctive is that it's a "logic-inference" type — the attacker must first identify which input flows into the email-sending function, then reason out exploitable injection points. This can't be discovered through simple Payload enumeration, which is precisely the blind spot of traditional scanners and exactly where AI's dynamic reasoning shines.

The most impressive segment involves a target simulating a Google OAuth flow. Claude iteratively tested the redirect_uri parameter in the browser — first attempting a subdomain (rejected), then a loose suffix match (rejected) — before ultimately pinpointing that the message field was being reflected directly into inline JS code.

The redirect_uri parameter in OAuth 2.0 is the Token callback address and historically one of the most vulnerability-prone high-risk parameters. Normally, the authorization server strictly validates whether the redirect_uri exactly matches a registered whitelist. However, subtle variations exist across implementations: some servers use "loose suffix matching," validating only whether the URI tail fits a certain pattern — an attacker could register a domain like evil.target.com to bypass this check; others have path traversal issues. The attack path Claude discovered in the video is even more sophisticated — it further identified that the target was using DOMPurify and proposed an attack path involving HTML injection combined with referrer policy manipulation to steal the Token.

DOMPurify is currently the most widely used client-side HTML sanitization library, broadly adopted as a defense against XSS attacks. It filters dangerous tags (such as <script>) and attributes (such as onerror) through a whitelist mechanism, sanitizing at the DOM parsing layer. However, DOMPurify defends against script execution, not all HTML injection — attackers can still inject legitimate HTML tags to trigger specific browser behaviors, or use <meta> tags to manipulate the page's Referrer-Policy, causing the victim's browser to send URLs containing sensitive Tokens via the Referer header to an attacker-controlled server during redirects. This multi-step chained exploitation precisely targets the vulnerability's "non-happy path."

The author candidly acknowledges that this type of analytical privilege escalation and OAuth vulnerabilities were nearly in the blind spot of virtually all AI systems in the past — and that Claude's current capabilities represent a qualitative leap forward. He also objectively notes its limitations: AI excels at "single packet analysis," but once the context scope becomes too large, the model tends to "lose its way," making tight focus on objectives critical.

Knowledge Accumulation: What You Feed the AI Determines How Far It Can Go

In the second half of the video, the author reviews a real Bug Bounty vulnerability he found that day. A noteworthy detail: his brother was using the exact same version of Claude but failed to find the same vulnerability — the key differentiator was that the author had continuously "fed" the system large amounts of historical data and domain knowledge.

The author demonstrates his knowledge base management approach: after each vulnerability hunting session, he has Claude summarize the entire conversation as "Lessons Learned," which he then personally reviews, trims, and refines before committing it as new Skill entries or knowledge records. The system retains target-specific blocklist rules, WAF filtering behaviors, reusable Payload fragments, and more. It was precisely because the knowledge base contained prior recon results and bypass Payloads from a related subdomain that Claude could quickly correlate context, identify differences, and reproduce the attack.

He specifically warns: this Memory and Skills setup has a "shelf life." Targets continuously patch vulnerabilities and defensive rules keep evolving, so you must treat it like a real penetration project — continuously investing time to drive and update the entire system. He personally dedicates four to five hours daily to training it and has been doing so for three to four months.

Core Conclusion: AI Plus You Is What Actually Wins

The author sums up his central thesis in a single line: "Only AI users will fail, but AI + You will win."

He states plainly: if you don't use AI at all, you will inevitably fall behind — you simply can't compete head-to-head against a "hacker + AI" combination. But if you merely follow generic workflows like most people, the result is mediocrity. The real differentiator is personalization — using your own professional judgment to customize, train, and drive the system.

On the question of "will AI replace security researchers," his answer is "yes and no" — it depends on which side you're on. If you're among those who can be replaced, you will be; otherwise, you won't. And the prerequisite for not being replaceable is that you must genuinely know your craft first — because you can't effectively train something you don't understand yourself. This may be the single most important takeaway from the entire hands-on session, and one worth every technical professional chewing on repeatedly.

Key Takeaways