Burning Through 11.7 Billion Tokens: Which Is the Strongest Cybersecurity AI Model?

A High-Stakes AI Cybersecurity Exam

As major vendors race to claim their large language models lead in coding, reasoning, and multimodal capabilities, a more specialized question often gets overlooked: In real-world cybersecurity scenarios, which AI model performs best?

To answer this question, a research team conducted a rather aggressive experiment — they consumed a full 11.7 billion (11.7B) tokens to systematically evaluate the cybersecurity capabilities of multiple mainstream large language models. What does that number mean? Tokens are the basic units LLMs use to process text — one English word typically corresponds to 1–2 tokens, while a Chinese character is roughly 1–2 tokens. Using 2024–2025 flagship model API pricing as a reference, input tokens for GPT-4o-class models cost approximately $2.50–$5.00 per million, with output tokens priced even higher. Estimating the 11.7 billion tokens at a blended input/output rate, the total cost likely fell in the $30,000 to $100,000 range, depending on how many models were evaluated and each model's pricing structure — and that doesn't include the human costs of developing the evaluation framework, preparing data, and analyzing results. This was a genuinely expensive experiment, but the motivation behind it deserves a closer look.

Cybersecurity is a field that demands extreme accuracy, domain expertise, and robustness. A model that scores well on general-purpose benchmarks won't necessarily excel at vulnerability analysis, attack chain reasoning, or secure code auditing. That's why dedicated, large-scale evaluations to verify a model's real capabilities carry significant practical value.

Why Cybersecurity-Specific AI Evaluations Are Necessary

The Limitations of General-Purpose Benchmarks

Widely used benchmarks like MMLU and HumanEval primarily measure a model's general knowledge and programming ability. MMLU (Massive Multitask Language Understanding), proposed by UC Berkeley and other institutions, is a large-scale multitask language understanding benchmark covering multiple-choice questions across 57 academic subjects — from humanities and social sciences to STEM — and is broadly used to gauge a model's breadth of general knowledge. HumanEval, released by OpenAI, is a code generation evaluation set containing 164 Python programming problems, assessing a model's code generation ability through test case pass rates. While both benchmarks are authoritative, they were never designed for adversarial thinking or deep reasoning in the security domain, creating inherent blind spots when evaluating a model's security analysis capabilities.

Cybersecurity is a highly specialized and adversarial field that requires models to possess a combination of the following capabilities:

• Threat identification: Accurately pinpointing security vulnerabilities in code or logs
• Attack reasoning: Understanding an attacker's mindset and simulating complete attack chains. The Kill Chain concept was originally proposed by Lockheed Martin, breaking down cyberattacks into seven stages: reconnaissance, weaponization, delivery, exploitation, installation, command and control, and actions on objectives. The MITRE ATT&CK framework later refined this into 14 tactical stages and hundreds of specific techniques, becoming the industry's common language for describing and classifying attack behaviors. Requiring an AI model to perform attack chain reasoning means the model must not only identify individual suspicious activities but also connect seemingly unrelated events to reconstruct an attacker's full intent and path — a complex cognitive task that heavily depends on contextual understanding and causal reasoning
• Defensive recommendations: Providing actionable, best-practice-aligned remediation plans
• False positive control: Avoiding excessive alerting or missing critical risks

These capabilities are difficult to measure through general-purpose tests. A model might generate code fluently yet fail to identify hidden SQL injection or buffer overflow vulnerabilities. SQL Injection is one of the most classic vulnerability types in web security — attackers embed malicious SQL statements in user inputs to bypass application authentication or directly manipulate databases. OWASP (Open Web Application Security Project) has listed it as the top web security risk for years. Buffer Overflow is a representative system-level security vulnerability — when a program writes data beyond a buffer's allocated space, attackers can overwrite adjacent memory regions, hijack program execution flow, and even gain system control. Many major security incidents throughout history, such as the 2003 SQL Slammer worm and the underlying exploitation mechanism of the 2017 WannaCry ransomware, are closely tied to these fundamental vulnerability types.

The Scale Behind 11.7 Billion Tokens

Consuming 11.7 billion tokens was no arbitrary decision. Building credible evaluation results requires repeated testing of each model across a large number of real or simulated security scenarios. This includes: multi-turn conversational penetration testing exercises, large-scale vulnerability sample analysis, and question bank coverage spanning multiple security sub-domains. Only a sufficiently large sample size can reduce randomness and yield statistically reliable conclusions.

This also reflects a broader trend: AI capability evaluation is shifting from "one-time benchmarking" to "large-scale, scenario-based stress testing." As model capabilities converge, simple benchmark scores can no longer differentiate quality. Deep domain-specific testing is becoming the new dividing line.

What This Experiment Means for the Security Industry

Practical Considerations for AI in the Security Operations Center

More and more enterprises are exploring the integration of large language models into their Security Operations Centers (SOCs) for alert triage, log analysis, and incident response. A Security Operations Center (SOC) is the central hub of an enterprise's cybersecurity defense system, typically staffed by a team of security analysts working 24/7 to monitor network traffic in real time, analyze security alerts, and coordinate incident response. A mid-sized enterprise's SOC may face thousands or even tens of thousands of alerts per day, the vast majority of which are false positives. Analysts must sift through massive noise to identify genuine threats. This high-intensity, highly repetitive work is precisely the scenario where large language models can deliver value — by automating alert triage and contextual correlation analysis, freeing analysts from initial screening tasks so they can focus on high-value deep investigations. Major security vendors including Palo Alto Networks, CrowdStrike, and Microsoft have already integrated LLM capabilities into their products, but industry consensus on model accuracy and reliability standards has yet to be established.

However, if model selection lacks rigorous evaluation evidence, the consequences can be severe — in the security domain, a single misjudgment or missed detection can cause substantial damage.

Therefore, these costly comparative evaluations are fundamentally about providing decision-making evidence for enterprise AI security selection. The question they answer isn't "which model is smarter" but "which model is most reliable for the security tasks I care about most."

Balancing Evaluation Costs Against Business Value

You might not have noticed, but the 11.7 billion token consumption also reveals a practical reality: deep evaluation itself is extremely expensive. For most enterprises, independently conducting tests at this scale simply isn't feasible. This also means the industry needs more open, transparent, and reproducible third-party security evaluation benchmarks to lower the overall barrier to model selection.

Conclusion: Vertical Evaluations Are Becoming AI's New Competitive Battleground

While the details of this "11.7 billion token" experiment still await more publicly available data for corroboration, the signal it sends is clear: As general capabilities approach saturation, deep evaluation in vertical domains is becoming the critical yardstick for measuring an AI model's true value.

For fields like cybersecurity, where the margin for error is extremely low, we need more than models that "look powerful" — we need models that "can withstand large-scale, real-world testing." In the future, we will likely see more specialized evaluations emerge for specific industries, and the results of these evaluations will speak louder than any marketing pitch.

Note: This article is based on limited source information. The specific evaluation methodology, list of participating models, and final rankings should be confirmed by following the research team's complete report.