Multi-Agent AI Detects Zero-Day CVE Exploitation: Technical Deep Dive Behind 85% Accuracy

Project Overview: Automated Zero-Day Detection with Multi-Agent AI

A noteworthy open-source project has emerged in the cybersecurity space — ai-detects-if-cve-was-zero-day. This project builds a multi-agent AI system that orchestrates GPT-4o, DeepSeek v3, and Llama 3.3 to work collaboratively, automatically detecting whether a given CVE vulnerability was ever exploited as a zero-day in the wild. On a validated set of 50 CVE samples, the system achieved over 85% detection accuracy.

Why Is Zero-Day Detection So Difficult?

Background

A zero-day vulnerability is a security flaw that has been exploited by attackers before the vendor releases a patch. The name comes from the vendor having "zero days" to fix the issue — the period between when an attacker discovers and exploits the vulnerability and when the vendor becomes aware and releases a patch is called the "zero-day window," during which all systems running the affected software remain unprotected. Zero-day vulnerabilities command extremely high prices on underground markets — a single iOS remote code execution zero-day can sell for millions of dollars on gray markets. Nation-state APT groups (such as Equation Group, APT28, etc.) are the primary consumers of zero-day exploits. Google's Project Zero team and Mandiant continuously track zero-day exploitation in the wild, with over 90 in-the-wild zero-days tracked in 2023 alone, showing a year-over-year upward trend.

Accurately determining whether a CVE was exploited as a zero-day is critical for threat intelligence analysis, security incident response, and risk assessment. Some background on the CVE numbering system is helpful here: CVE (Common Vulnerabilities and Exposures) is a globally unified vulnerability identification system maintained by MITRE Corporation, formatted as CVE-Year-Number (e.g., CVE-2024-3400), with each entry recording the vulnerability's basic description, affected products, and version information. NVD (National Vulnerability Database) is maintained by the National Institute of Standards and Technology (NIST) and adds structured data such as CVSS scores, CWE classifications, and CPE matches on top of CVE entries. However, neither CVE nor NVD systematically labels whether a vulnerability was exploited as a zero-day — this information is typically scattered across security vendor threat reports, CISA's KEV (Known Exploited Vulnerabilities) catalog, and various threat intelligence platforms.

Traditionally, this determination process has been heavily dependent on manual analysis by security researchers. Analysts must sift through numerous information sources — vulnerability advisories, security vendor reports, attack timelines, dark web discussions — making the entire process time-consuming and prone to missing critical evidence.

The Project's Automated Approach

This project automates this complex analytical workflow through a multi-agent architecture, decomposing the judgment task into multiple sub-tasks handled by different AI models:

• Forensic Evidence Extraction: Gathering exploitation timelines and evidence from public sources
• Pattern Recognition: Identifying characteristic features and behavioral patterns of zero-day exploitation
• Threat Intelligence Integration: Cross-validating and making final determinations by synthesizing multi-source intelligence data

Multi-Agent Technical Architecture Deep Dive

Technical Principles of Multi-Agent Systems

Multi-Agent Systems (MAS) are a classic architectural paradigm in artificial intelligence that have experienced a renaissance with the maturation of large language models. In LLM multi-agent frameworks, each agent is an LLM instance with a specific role, instructions, and tool-calling capabilities. Representative frameworks include Microsoft's AutoGen, CrewAI, and LangGraph. The core advantages of multi-agent systems over single models come in three areas: First, task decomposition — breaking complex problems into multiple sub-tasks where each agent focuses on what it does best; Second, adversarial verification — different agents can challenge and verify each other, similar to academic peer review, effectively reducing LLM hallucination issues; Third, diversity — agents using different underlying models have different knowledge biases and reasoning styles, and combining multiple perspectives improves judgment robustness.

Division of Labor Among Three Models

The project selected three large language models with complementary strengths, forming a synergistic analytical system:

Model	Core Strength	Likely Role
GPT-4o	Strong comprehensive reasoning	Overall judgment and coordination
DeepSeek v3	Outstanding logical reasoning	Evidence chain analysis
Llama 3.3	Open-source and controllable	Pattern matching and verification

From a technical characteristics perspective, GPT-4o is OpenAI's multimodal flagship model released in 2024, with "o" standing for "omni," performing at the top level in text reasoning, code generation, and multimodal understanding, with its massive training data providing rich knowledge in the cybersecurity domain. DeepSeek v3 is an open-source large model released by DeepSeek, employing a Mixture of Experts (MoE) architecture with 671B total parameters but only 37B activated per inference, delivering outstanding performance on mathematical reasoning and logical analysis benchmarks at inference costs far below comparable closed-source models. Llama 3.3 is Meta's open-source model with 70B parameters, excelling in multilingual understanding and instruction following. The strategy of selecting these three models reflects a "closed-source + open-source" and "commercial + free" balance, ensuring analysis quality while reducing dependency risk on any single vendor.

The core advantage of the multi-agent system lies in different models analyzing the same CVE from different angles, using "voting" or "debate" mechanisms to reduce individual model bias and hallucination, thereby improving the reliability of final judgments. LLM "hallucination" refers to models generating content that appears plausible but is actually incorrect or fabricated — particularly dangerous in security analysis scenarios where models might fabricate non-existent exploitation evidence or incorrectly attribute timeline information. The multi-model voting mechanism works similarly to ensemble learning in machine learning: the probability of multiple independent models making the same error is far lower than a single model making an error. Common voting strategies include majority voting, weighted voting (assigning weights based on each model's historical accuracy on specific tasks), and cascading verification (one model's output serves as input for another model's secondary verification).

CVE Zero-Day Detection Workflow

Based on the project description, the system's workflow is roughly divided into five phases:

1. Input CVE ID → Automatically collect public information from NVD, security advisories, etc.
2. Forensic Analysis → Extract key time points including vulnerability disclosure date, patch release date, and known exploitation date
3. Pattern Matching → Compare extracted features against typical patterns of known zero-day vulnerabilities
4. Multi-Model Voting → Three AI models each provide their judgment, synthesizing a final conclusion
5. Output Results → Generate an analysis report including reasoning basis and confidence level

85% Accuracy: Achievements and Limitations

Practical Significance of the Accuracy Rate

Achieving over 85% accuracy on 50 validated CVE samples needs to be understood from two dimensions:

• Practical Value: For assisting security analysts with rapid screening and vulnerability prioritization, 85% accuracy can already significantly improve work efficiency
• Real-World Limitations: A test set of 50 samples is relatively small, and "validated" samples may have selection bias; accuracy could fluctuate in larger-scale, more complex real production environments

From an evaluation methodology perspective, 50 samples falls at the Proof of Concept level. Achieving production-grade reliability requires considering several key dimensions: First, sample balance — zero-day vulnerabilities represent an extremely small fraction of all CVEs (fewer than 100 confirmed zero-day exploitations out of approximately 100,000 CVEs per year), so the ratio of positive to negative samples in the test set significantly affects accuracy interpretation; Second, temporal generalization — model performance on historical CVEs may not generalize to newly disclosed vulnerabilities; Finally, choice of evaluation metrics — in such imbalanced classification problems, Precision, Recall, and F1 Score are more informative than raw Accuracy. For example, if the system tends to misclassify many CVEs as zero-days (high recall but low precision), while it won't miss actual zero-days, it will generate numerous false positives that increase analyst workload.

Suitable Application Scenarios

• Threat Intelligence Automation: Quickly flagging CVEs with potential zero-day exploitation to reduce response time. Threat Intelligence refers to security threat information that has been collected, processed, and analyzed. It can be categorized by consumption level into strategic, tactical, and operational intelligence. Major threat intelligence platforms include Recorded Future, Mandiant Advantage, and ThreatBook, with STIX/TAXII as the industry-standard sharing protocol.
• SOC Operations Support: Helping security operations teams prioritize high-risk vulnerabilities and optimize resource allocation. A SOC (Security Operations Center) is the central hub of enterprise cybersecurity defense, where security analysts monitor security alerts 24/7. SOCs currently face severe "alert fatigue" — a SANS Institute survey shows that over 70% of SOC teams report alert volumes exceeding their processing capacity, with organizations potentially receiving thousands or even tens of thousands of security alerts daily, many of which are false positives. Therefore, AI-assisted vulnerability prioritization and threat assessment tools address enormous real-world demand.
• Security Research Tool: Providing researchers with preliminary screening capabilities to reduce repetitive labor

Project Status and Future Directions

The project currently has 20 stars and 8 forks on GitHub, developed in Python. While community attention is still in its early stages, its core concept is quite inspiring — using multi-agent AI systems to handle complex judgment tasks in cybersecurity.

Possible future improvements include:

• Expanding the Validation Dataset: Covering more CVE types and years to improve statistical significance
• Enriching Information Sources: Integrating dark web intelligence, PoC code analysis, social media monitoring, and other data
• Optimizing Collaboration Strategies: Improving voting weights and debate mechanisms between models
• Adding Real-Time Capabilities: Supporting real-time monitoring and automated analysis of newly disclosed CVEs

Conclusion

The ai-detects-if-cve-was-zero-day project demonstrates the application potential of multi-agent AI in cybersecurity analysis. By orchestrating GPT-4o, DeepSeek v3, and Llama 3.3 to work collaboratively, it partially automates the zero-day exploitation determination task that traditionally requires senior security analysts spending considerable time. Although still at the proof-of-concept stage, the 85% accuracy rate already validates the feasibility of this technical approach. As model capabilities continue to improve and data sources expand, this "AI-assisted security analysis" paradigm deserves ongoing attention from security professionals.