Evaluation Hallucinations: 19 Attack Families Registered, Only 3 Ever Tested Against Real Models

When "Can Express" Gets Mistaken for "Has Been Verified"

A developer maintaining a red-teaming tool for robotic VLA (Vision-Language-Action) policies shared a sobering self-audit on Reddit. He took a fresh-eyes look at his own tool's registry, then compared it against the file that actually recorded results — and stumbled into a trap that nearly everyone doing evaluations eventually falls into.

The registry listed: 19 attack families, 44 attack variants, 8 strategies. Sounds impressive. But the file actually documenting measured results contained just 30 rows of data — 25 of which were the same strategy against the same test set, and 5 were "stub against stub" (placeholder against placeholder). Beyond that, not a single row covered any other strategy. Of the 19 registered attack families, only 3 had ever actually been run against a real model.

VLA (Vision-Language-Action) policies represent a cutting-edge paradigm in robotics, unifying visual perception, natural language understanding, and robot action control within a single model framework. "Red teaming" — borrowed from cybersecurity — refers to deliberately crafting adversarial inputs or scenarios to expose failure modes, out-of-bounds behavior, or safety vulnerabilities. Because VLA models must execute actions in real or simulated physical environments, red-teaming them is far more complex than testing pure language models — it involves not just semantic-level attacks but also visual perturbations, instruction ambiguity, and physical constraint violations. This is precisely why "number of attack families" is often used as a proxy metric for red-team tool coverage — and also precisely why it sets the trap described here: attack types you can categorize and describe are a very different thing from attacks you've actually run and recorded results for.

Two Files Answering Two Different Questions

The author identified the core issue: these two files diverge because they're answering fundamentally different questions.

The registry answers "what can this thing express" — it describes capability boundaries, design intent, and theoretical coverage. The measurement file answers "what has this thing actually been pointed at" — it records what has genuinely been validated.

The critical distinction: only the second file constitutes evidence. Yet ironically, what ends up in the README and gets promoted externally is almost always the first file's impressive-sounding number. "19 attack families" is far more persuasive than "3 actually tested," and so the gap between the marketing number and the validation number is born.

This isn't fraud — it's a subtler form of self-deception. When you repeatedly look at the capability checklist in your registry, it's easy to mistake "I could do this" for "I have done this."

A Pragmatic Suggestion: Let a "Freshness File" Publicly Call You Out

The author's solution shows real engineering wisdom: place a freshness file next to the headline numbers you publish externally, recording which version those numbers were measured on — and let it publicly contradict you.

His own freshness file currently shows: measurements taken at version 0.32.0, while the current version is 0.41.2 — nine full versions behind. Even more telling is his candid admission: this file has been showing this discrepancy for about a week, and every time he's seen it, he's scrolled right past it.

This actually proves the mechanism works — the automated drift detection is doing its job; it's the human who's slowed down. The system faithfully tells you the data is stale, but human inertia renders the warning ineffective. This observation is immensely valuable for anyone building evaluation infrastructure: mechanisms can expose problems, but they can't force you to act. That's why the freshness file must be public — letting external pressure compensate for human laziness.

Version drift is a common problem in software engineering, and particularly dangerous in evaluation contexts. When model weights, inference frameworks, or test environments change, the validity of historical measurements gradually decays — but this decay is typically invisible. No error, no warning, just a timestamp that keeps getting older. A "freshness file" is essentially an explicit expiration declaration mechanism, similar to a "best by" date on food packaging — it converts invisible decay into a visible status gap. This approach has parallels in continuous integration (CI) practice, such as embedding "metric validity assertions" into pipelines that automatically trigger warnings or block releases when the gap between current and measured versions exceeds a threshold. The author's choice to make freshness information public rather than keeping it internal reflects an engineering philosophy of institutionalizing external accountability pressure — when personal discipline fails, social pressure picks up the slack.

Thirty Cents for One Real Measurement

At the end of the post, the author mentions that today he ran the second strategy for the first time: ten episodes, at roughly thirty cents in GPU cost.

This detail is worth dwelling on. What blocks real validation is rarely cost — thirty cents is nearly negligible — it's procedural inertia and the illusion that "registering equals completing." When backfilling a single real measurement is this cheap, yet it keeps not happening, the problem clearly lies in priorities and habits, not resources.

He closes by throwing a question to the community that everyone doing benchmarking and evaluation should sit with: How do you prevent your published numbers from drifting away from your actually-measured numbers?

Lessons for Evaluation Engineering

This self-reflection from a single developer touches on a universal pain point in AI evaluation. As models and tools iterate rapidly, any static "capability claim" quickly falls out of sync with actual validation status. Here are a few immediately actionable practices:

• Distinguish capability lists from evidence lists: Numbers in your README should come from measurement files, not capability registries.
• Version-bind every metric: Every publicly reported number should be annotated with the version it was measured on.
• Let the system publicly contradict you: Freshness detection shouldn't be hidden — make it visible to both your team and your users as a kind of institutional conscience.
• Reduce the friction of verification: When the cost of a backfill test is as low as thirty cents, there's no justification for letting claimed numbers hang in the air indefinitely.

The value of an evaluation tool ultimately lies not in what it "claims" to be able to measure, but in what it has actually measured. This post reminds us: in an era of rapid AI iteration, the first step toward honesty is building systems whose numbers are willing to publicly call themselves out.