aegis-skills Open Source Project: 40 Defensive Security Audit Skills Explained

An Unexpected Model Experience

As AI coding tools become increasingly mainstream, "vibe coding" has become a daily reality for many developers. Yet this rapid-iteration approach often introduces hidden security vulnerabilities. Recently, a Reddit developer shared their experience of systematically distilling security audit skills with the help of an AI model while building a project that required sandbox isolation — sparking a community discussion about the viability of AI-assisted security auditing.

The developer's initial struggle is a familiar one: when using a certain AI model (referred to in the post as "Fable 5"), simply asking basic questions about sandboxing and feature isolation was enough for the model to flag the request as "hacking" and refuse to respond. This overly conservative safety alignment actually became an obstacle to legitimate security research. Constrained by hardware limitations, the author could only run local models up to 30B parameters, further narrowing their options.

Distilling Security Audit Knowledge from Ox Alpha

The breakthrough came when the author tried a newly released mystery model called "Ox Alpha." According to their account, the model's coding assistance exceeded expectations — even surpassing their previous experience with Opus 5. More importantly, during its current free and relatively unrestricted window, Ox Alpha was willing to provide substantive answers on security audit topics.

The author seized this opportunity and decided to "distill" the model's outputs into a reusable skill set. Here, "distill" isn't used in the strict machine learning sense, but rather as a metaphor: extracting the security audit experience, checklists, and best practices surfaced across multiple conversations into structured, actionable skill modules.

aegis-skills: 40 Defensive Security Audit Skills

The result is an open source project called aegis-skills (hosted under the AS-FOSS organization on GitHub), containing 40 defensive security audit skills. These skills focus on "defense" rather than "offense," with the goal of helping developers identify and patch the obvious security vulnerabilities that commonly arise during rapid development.

For developers who rely heavily on AI-generated code, the value of such a skill set lies in providing a systematic self-audit framework — compensating for the security review experience that non-specialist developers often lack.

A note on "Model Distillation": In machine learning, "model distillation" formally refers to training a smaller student model using the outputs of a larger teacher model, preserving capability while reducing size. The author borrows this term as a metaphor for a "knowledge distillation" process — using systematic multi-turn conversations to make a large model's implicit domain expertise explicit and structured, ultimately producing checklists and operational guidelines that humans can use directly. In essence, this treats the model as a "knowledge extractor" rather than a direct code generation tool — conceptually closer to traditional expert interviews and knowledge engineering, except the "expert" being interviewed is a language model.

Analyzing the Limitations of This Skill Set

The author is refreshingly candid about the limitations of this skill set — a honesty that itself deserves acknowledgment.

The Author Is Not a Security Expert

The project creator explicitly states they are "neither a security expert nor someone who has done extensive comparisons with models like GLM 5.2." This means the skill set is fundamentally a secondary organization of AI model knowledge, whose accuracy and completeness depends heavily on the underlying model's output quality — without cross-validation from authoritative security professionals.

Can It Actually Patch Vibe Coding Vulnerabilities?

The author's core question is a genuinely practical one: Can these skills at least patch the most obvious "vibe-coded" vulnerabilities? This touches on the fundamental premise of AI-assisted security auditing — can AI-generated security advice effectively guard against the flaws in AI-generated code itself?

There's a slightly circular logic here. Using the same class of AI tools to both generate and audit code may create a "shared blind spot": models may lack the ability to recognize the kinds of mistakes they themselves tend to make. For this reason, skill sets like this are better suited as a supplementary tool rather than a replacement for professional security audits.

What is "Vibe Coding"? The term was coined by AI researcher Andrej Karpathy in early 2025 to describe a style of development that relies heavily on AI models, where developers build software rapidly with little deep understanding of the underlying code. Its defining characteristics: describe requirements in natural language, accept AI-generated code blocks, and ask the AI to fix any errors that arise — all without manually reading code in detail. This approach dramatically lowers the barrier to building software, but introduces systemic security risks. Because developers lack an understanding of the code logic, they also lack awareness of potential vulnerabilities. Common issues include unvalidated user input passed directly into database queries (SQL injection risk), hardcoded keys or credentials, and API endpoints lacking permission checks. This is precisely the gap that audit skill sets like aegis-skills aim to fill.

Risks Around Model Availability and Sustainability

The author specifically mentions wanting to distill knowledge "while Ox Alpha is still free and unrestricted." This hints at an underlying concern: the sustainability of knowledge distilled from a specific model — especially one in a testing phase with unsettled policies — is questionable. If the model tightens its restrictions or goes offline, updating and maintaining the associated knowledge base will become a significant challenge.

Industry Implications for AI-Assisted Security Auditing

This case, though small in scale, reflects several important trends and tensions in the current AI coding ecosystem.

First, the "over-refusal" problem in AI safety alignment is becoming increasingly prominent. When models treat legitimate topics like security research and sandbox design as potential threats, they actively impede defensive security work. How to balance preventing misuse with supporting legitimate research is a direction that model providers will need to continuously optimize.

Second, community-driven knowledge distillation is emerging as a new form of open source practice. Developers are transforming AI model capabilities into reusable, reviewable, and collaboratively improvable open source assets — a pattern that helps knowledge accumulate and spread, and opens the door for broader participation in verification and refinement.

Finally, for the many developers using AI-assisted programming, the mindset of proactively conducting security self-audits matters far more than any specific tool. Regardless of how effective a skill set like aegis-skills ultimately proves to be, it at least reminds developers of a fundamental truth: AI-generated code cannot be assumed to be secure by default, and systematic audit habits must be built.

On AI "Over-Refusal": Over-refusal (also called over-alignment) refers to a phenomenon where models, in the course of safety training designed to prevent misuse, mistakenly flag large numbers of legitimate technical queries as harmful and refuse to answer them. Topics essential to security research — sandbox design, permission isolation, vulnerability reproduction — often become collateral damage because they share vocabulary with offensive techniques. This problem exists in both open and closed-source models, though it manifests differently: closed-source models may refuse outright, while open-source models can typically be worked around via system prompts or fine-tuning. Over-refusal not only reduces a tool's practical value for security professionals, but can also create a false sense of security — leading users to mistakenly believe that certain topics are inherently dangerous, and thus avoid necessary security discussions.

Conclusion

Starting from a personal project roadblock, using an emerging AI model to distill 40 defensive security audit skills and releasing them as open source — this is, in its own way, a snapshot of developer creativity in the age of AI. Even though the author humbly acknowledges they're not an expert, and the results still await community validation, this practice of "learning by doing and contributing along the way" is a vivid expression of the open source ethos.

For readers interested in AI programming security, it's worth treating skill sets like this as a starting point and a basis for discussion — not as a definitive answer. In an era where AI is deeply involved in code production, how to keep security auditing pace with the speed of generation remains an open question that the entire community must explore together.