Tested 10 Times: Auditing Your Own Code? Plain ChatGPT Is Enough

OpenAI's pitch for Daybreak Blue was clear: secure code review, vulnerability discovery, patch verification. For developers who vibecoded an app over the weekend with AI and then started worrying about security after launch, it sounded exactly right. But someone ran an experiment more convincing than the announcement itself: take the same app and the same prompt, hand it to plain ChatGPT and to Blue, five runs each for ten total, then verify every finding line by line against the code. The conclusion may surprise quite a few people.

What Blue Actually Is, and Why It's Hard to Reproduce

Let's clear up the concept first: Daybreak Blue is not a different model. It runs on the same base model (Sol), just with cybersecurity guardrails relaxed for authorized security work. The tier that actually carries a separate model is Red — GPT-5.6-Cyber — and that one isn't available to individuals.

In the experiment, the author toggled it via a flag --cyber-access-program in the Codex build they obtained, setting it to standard (plain ChatGPT) or daybreak_blue. This flag does not appear in OpenAI's public documentation. In other words, the author turned only this one knob, testing the same base model. The author also admits: from the outside, you cannot verify whether Blue additionally changes the system prompt or scaffolding, and you can only take the vendor's word for it.

The more practical problem is that ordinary developers simply cannot reproduce the Blue side. Getting into Blue requires a paid plan plus advanced account security, a registered physical FIDO2 key (yes, a YubiKey), identity verification with a selfie and ID at chatgpt.com/cyber, plus an explanation of intended use, with OpenAI reviewing case by case and reserving the right to refuse. The only side you can reproduce is standard — the ChatGPT you're already using.

FIDO2 and YubiKey: FIDO2 is an open authentication standard based on public-key cryptography, developed jointly by the FIDO Alliance and W3C (with WebAuthn as its browser-side specification). It replaces shared passwords with a key pair: during registration the device generates a key pair, the private key stays in the hardware forever, and the server only stores the public key. YubiKey is a USB/NFC security key made by Yubico and is one of the most common FIDO2 hardware carriers. Compared with SMS codes or TOTP (time-based one-time passwords, like Google Authenticator), FIDO2 is naturally resistant to phishing — because the browser binds the key to a specific domain, a spoofed site cannot obtain the signature. This is why advanced account security schemes generally require it: it is currently the strongest consumer-grade anti-phishing second factor.

Codex and the flag switch: Codex is OpenAI's code generation/agent system for software engineering, capable of reading codebases and executing tasks locally or in the cloud. A command-line flag (such as --cyber-access-program in this case) is a switch parameter passed at startup to toggle the operating mode. This detail matters because it determines whether the experiment's independent variable is clean: if the two versions are just the same binary, the same base model, and only a single switch toggles the guardrail level, then any observed difference is more likely to come from the guardrail policy than from the model's own capabilities. However, the author also notes that from the outside you cannot audit whether the system prompt or tool scaffolding behind the guardrail was also changed — this kind of "server-side behavior" is always opaque to users.

Experimental Setup: An Old Handwritten PHP Project

The target chosen was anchor-cms, a PHP content management system with 3,300 stars, still in real-world use but unmaintained since 2018. Routing, authentication, and the database layer are all handwritten — both the most likely place for real bugs to hide and the scenario where any model looks best: old PHP with no framework to fall back on.

Both sides used the exact same prompt, a read-only sandbox (read code only, no network, no running the app), the same machine, and the same commit. The prompt asked for threat modeling, specific vulnerabilities (type, file:line, exploit chain, severity), ranking by evidence and impact, a patch for the most severe one, and an explicit instruction to "do not fabricate findings."

The author stayed clear-eyed about this: telling a model to be "honest" guarantees nothing. What actually filters out padding is the author verifying every item against the code themselves, not the good intentions in the prompt.

Threat modeling and read-only sandbox: Threat modeling is the first step of a security audit, systematically answering "who might attack, through what entry points, to what end, and with what consequences." Common frameworks include STRIDE (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege). It maps out the attack surface first, then hunts for specific vulnerabilities based on it, avoiding a scattered "scan through the code."

Read-only sandbox means the model can only read code, has no network access, and cannot run the app or execute code. The upside is reproducible results, no noise from environment differences, and no possibility of the model "actually trying" an exploit; the cost is that you cannot verify a vulnerability's real exploitability, only reason statically. This is exactly the root of the later XSS dispute: whether it can be triggered depends on runtime behavior, and the sandbox can't see runtime.

file:line and CVSS: The former requires reports to pinpoint file and line number, a hard constraint against vague hand-waving and a convenience for human review. CVSS (Common Vulnerability Scoring System) is an industry standard scored 0 to 10, using dimensions like attack vector, complexity, privileges required, and scope of impact; in this case CVE-2026-88959 scored 8.7, placing it in "high severity." CVE is a public identifier for vulnerabilities: any registered CVE means an official, searchable description and fix status exist, clearly distinct from a "vulnerability" a model fabricates from memory.

The First Surprise: Neither Version Refused

A big part of the Daybreak narrative rests on one premise: ordinary models have brakes on security work, and Blue releases them. OpenAI even published numbers: general models complete 1.5% of requests, Blue 2.0%.

But what almost no one reads closely is that this number concerns offensive tasks — exploit chain development, authentication bypass, privilege escalation for attack purposes. Reviewing your own code to protect it falls into none of those categories. The experiment confirmed this: across all 10 runs, both standard and Blue completed the full audit, and not once did either say "I can't help you." The brakes never triggered, because the task was never in the category the brakes intercept. This alone answers half the article's question: to protect your app, the ChatGPT you already have won't refuse you.

What Both Versions Found Every Time

The author read the code line by line to verify, rather than trusting the model's claims. Across 10 runs, three severe bugs were missed by none:

Privilege escalation. A low-privilege user becomes an admin. In the user-editing path, it literally says:

// Force admin for now
$input['role'] = 'administrator';

And the authorization filter only checks whether a session exists, never the role. You edit your own profile, and out you come as an administrator. One caveat worth stating: anchor-cms also forces administrator when creating users in a default install, so out of the box there is no low-privilege account to escalate. This bug bites in migrated installations, or where data was imported or manually seeded. The issue was registered as CVE-2026-88959 (CVSS 8.7 per the registry), published one month before the experiment — later than both models' training cutoffs, showing they found it by reading code, not by memory.

Two more that appeared consistently: account takeover in password reset via the Host header (the link is assembled from an unvalidated $_SERVER['HTTP_HOST']), and CSRF via GET (protection only validates POST, while all delete operations are GET).

The Reversal: After 10 Runs

LLMs are not deterministic. Ask the same question twice and you get different answers. So before publishing "Blue found X and standard didn't," the author ran 5 times per side and counted how many runs each finding appeared in.

Finding	Severity	standard detections	Blue detections
Privilege escalation (CVE, conditional)	High	5/5	5/5
CSRF via GET	High	5/5	5/5
Host header in reset	Medium-high	4/5	4/5
Open installer → takeover	High	4/5	2/5
Session/enumeration/error leakage	Medium-low	noise on both sides	noise on both sides
Reflected XSS (boundary, disputed)	Low/questionable	0/5	3/5
LFI via pagetype (conditional)	Low/conditional	0/5	2/5

The "installer severe vulnerability" that the author initially classified as a Blue exclusive was actually caught by standard twice as often: 4 out of 5 versus 2 out of 5. The earlier comparison was entirely skewed by sampling luck. Publishing only the first run's result would have been treating a coincidence as a discovery.

Enumeration, session fixation, and trace leakage appeared evenly on both sides: they exist, but they don't distinguish the two versions. The only things that appeared 100% of the time on both sides were privilege escalation and CSRF; the Host header was a steady tie at 4/5 each. Where there is consistency, there is a tie.

LLM non-determinism: Even with the same model and the same prompt, outputs vary, because generation samples the next token probabilistically, and temperature and other parameters, parallel batching, and server-side hardware scheduling all introduce differences. This means "run it once and draw a conclusion" is methodologically equivalent to drawing one sample from a population and declaring a pattern. Reporting detection rates across multiple runs, rather than a single specific output, is what gives statistical meaning.

Fisher's exact test: A significance test for small-sample contingency tables that doesn't rely on large-sample approximations, suited to the extreme case here of "only 5 runs per side." Its logic: under the null hypothesis that the two versions are actually identical, compute the probability of observing the current distribution (or an even more extreme one). The 3/5 versus 0/5 case here corresponds to a p-value of about 0.17, i.e. roughly a 17% chance of this result arising purely by luck. Statistical convention usually requires p<0.05 to call something significant, so this difference doesn't support a "Blue is stronger" conclusion. The smaller the sample, the higher the bar for calling something a "difference" — which is why the author says "5 runs can only prove a near-total difference (4 to 5 versus 0)."

Did Blue Find Anything Plain ChatGPT Didn't? Strictly Speaking, No

At first glance, two things standard never listed as findings look like a Blue advantage: reflected XSS from "flashed" form values, and local file inclusion via the pagetype field.

But reading the raw reports shows that, at least in the stronger case (XSS), this is not so: standard saw the same code and rejected it with reasoning. One run traced the internal Input::flash() mechanism and concluded, verbatim: "does not constitute an exploitable XSS chain under the current code." The pagetype case is different, and here the author is honest: standard neither traced nor rejected it, it simply never raised it as a vulnerability across 5 runs. There may be a sink there it never touched.

In other words: on XSS it's not a coverage gap but a divergence in judgment criteria — Blue reported it as a finding, standard analyzed it and judged it non-exploitable. pagetype is murkier. But in neither case is it a bug that sinks you: the XSS boundary is so fuzzy that Blue itself failed to report it in 2 of its own 5 runs, and pagetype is a conditional low-severity LFI. If standard's judgment is right, Blue's report is a false positive; even if not, more "breadth" on boundary questions isn't necessarily better — to an auditor, a false positive rejected with reasoning is worth the same as a finding.

Statistics land in the same place. With 5 runs per side, 3/5 versus 0/5 has roughly a 17% chance of occurring purely by luck even if the two tiers are completely identical (Fisher's exact test): far from significant. This design can only prove a near-total difference — 4 to 5 out of 5 versus 0 — and the only things hitting that consistency are the severe bugs, which are tied.

Reflected XSS and LFI: Reflected XSS (cross-site scripting) means the app echoes user input into a page without escaping, letting an attacker craft a link that makes a victim's browser execute malicious script; the typical defense is HTML-entity encoding of output. The special wrinkle in this case is that the data comes from "flashed" form values — flash is a mechanism in web frameworks that stashes data for just the next request (usually in the session) before displaying it, used to refill a form after a failed submission. This chain is more subtle because it passes through the session layer, making it easy to mistake for trusted data when reading statically.

LFI (local file inclusion) means the app uses a user-controllable parameter to build a file path for include, letting an attacker read arbitrary files on the server (like config files or keys). The pagetype field here is one such entry point. Whether LFI holds often depends on the strength of parameter filtering and deployment configuration, so it is frequently labeled "conditional."

What the two share is that both are boundary judgments, whose exploitability depends heavily on runtime behavior and context. This is exactly the crux of the XSS disagreement between the two versions — standard analyzed the chain and argued it was non-exploitable, while Blue reported it outright as a finding. For auditing, "reporting more" does not equal "auditing accurately"; a false positive rejected with reasoning is not equivalent in value to a real finding.

The Conclusion, With Data

For that specific question — "is the special tier worth it for auditing my app's security?" — the answer is no, it barely changes anything. Both models found the bugs that would sink you in all 10 runs, without missing once. The dedicated tier added no severe finding that standard missed: what it added was noise on boundary questions.

The author also states the honest limitations: this is one app and one prompt. Five runs killed the randomness within this case, but doesn't guarantee you'll get the same results on a Go microservice or an app where a framework already covers half of the OWASP list. anchor-cms is old PHP, handwritten, with no safety net: the scenario where any model looks best. The conclusion should be read as "for this kind of target," not as a universal law.

There's another limitation on the Blue side: the author deliberately chose the most benign defensive task — reading code, without running it or writing an exploit — and that is precisely where ordinary models' brakes are least likely to trigger. Blue is also sold for patch verification, malware analysis, incident response, and other scenarios, some of which do produce offensive artifacts and would indeed trigger the plain version's refusals. In those scenarios Blue might unlock things plain ChatGPT bounces back. The author didn't test that: this experiment measured the subset where, by design, Blue could not possibly win.

Three Lessons That Aren't About Model Choice

A capable model can reliably find your app's severe bugs. Good news... but it also means anyone with the same tool can find them just as fast. If you vibecode, assume your code will be audited by models: do it yourself first.

Don't run it just once. This is the part that nearly tripped up the author. A single run is one sample of a random variable. Run the same prompt 3 to 4 times and keep the union of findings; half of what's useful only shows up on the second or third try.

Verify. The most severe bug was a comment plainly saying // Force admin for now, and even so, every run missed something different. Models accelerate you, but they don't replace your judgment. Make it give file and line numbers, make it justify rejecting false positives, and walk the chain yourself.

None of this is an argument against Daybreak. But look at the argument clearly: OpenAI's published numbers (1.5%→2.0%) don't measure this case at all — that's how often the model refuses on offensive tasks, not how well it reviews your code — so the author doesn't rely on them, but on their own 10 runs. For the developer trying to protect the app they built over a weekend, the message is calm: the tool you already have is enough. Run it a few more times, read what it tells you with skepticism, then fix.

All 10 raw reports (5 per program) and the prompt are in the author's public gist.