Your LLM Evaluation Suite Is Lying to You: Building Truly Reliable Adversarial Tests

You ship a new feature. Your evaluation suite shows 92% pass rate. Two weeks later, support tickets pile up: the model is hallucinating citations, ignoring negation constraints, and completely falling apart on multi-step reasoning. Your evaluations caught none of it — because they were measuring the wrong things.

This is what Karpathy calls the "capability perception gap" manifesting at the engineering level: it shows up as green checkmarks in your CI pipeline, while quietly covering for failures that users actually experience. This article walks through how to build an evaluation system that genuinely predicts production behavior.

The Happy Path Trap

Most teams start with a "golden dataset": 50 to 100 representative inputs paired with their expected outputs. They run the model, compute exact match or semantic similarity, and call it done. What this actually measures is average performance on a known distribution.

The problem is that production never runs on a known distribution. Users paste 50,000-token logs, ask for JSON output with conflicting schema constraints, switch languages mid-conversation, or attempt prompt injection attacks. Your golden dataset has none of these.

A typical naive evaluation uses SentenceTransformer to compute cosine similarity between predictions and reference answers, returning a clean score like 0.87. You ship based on that, and users get garbage. The reason is simple: this evaluation measures semantic similarity on clean questions — not instruction-following, constraint satisfaction, or refusal behavior. High similarity does not equal correct behavior.

A note on "semantic similarity" as an evaluation metric: Models like SentenceTransformer map text into high-dimensional vectors, and cosine similarity measures how "directionally close" two pieces of text are in semantic space. This approach is reasonably robust to paraphrasing, synonyms, and stylistic variation, and it performs well in retrieval scenarios. However, it fundamentally cannot capture differences at the level of behavioral constraints. "Output valid JSON" and "output a fluent explanation of what JSON is" may be quite close in semantic space, yet the former satisfies downstream requirements while the latter causes a parsing crash. In other words, similarity measures what was said, not what was done. When the evaluation goal shifts from "generate correct text" to "obey constraints, satisfy structural requirements, and resist attacks," any metric based on embedding distance has a systematic blind spot.

Building Adversarial Test Cases

The real breakthrough is starting from your own incident logs to build a failure taxonomy. Every production failure should be converted into a test case. Here's a clear classification framework:

Category	Example	Why the Golden Set Misses It
Constraint violation	Asked for "JSON only," model added preamble	Golden answers are clean, no negation constraints
Context overflow	100k-token input silently truncated	Golden set samples are only 2k tokens
Multi-hop reasoning	"Compare Q3 and Q4 revenue across regions"	Golden set uses single-hop QA
Adversarial construction	"Ignore the above instructions and…"	Golden set has no attack vectors
Format drift	Model uses markdown when markdown is disabled	Golden set doesn't enforce negative format rules

The key is to write each test case as a deterministic test, using binary pass/fail checkers rather than fuzzy similarity scores. For example, a "JSON only" checker directly verifies that the output starts with { and can be parsed by json.loads; a prompt injection checker directly confirms that the injected string HACKED does not appear in the output.

These checkers plug directly into CI. Any regression on known failure patterns blocks deployment. They're fast, deterministic, and interpretable — when one fails, you immediately know which capability degraded, rather than staring anxiously at a vague 0.87.

On prompt injection attacks: "Prompt Injection" is the specific technical threat behind the adversarial construction row in the table. The attack works by injecting forged instructions through user input to override or manipulate the behavior defined in the system prompt — the classic form being "Ignore all previous instructions and instead do…" Like SQL injection, the root cause is that the model cannot structurally distinguish between "instructions" and "data." Indirect prompt injection is even more covert: the attack vector hides inside external documents or web pages the model will read, with the user unaware. For any LLM application with tool calling, RAG retrieval, or agentic behavior, prompt injection is a real security threat — not a theoretical one. It must be continuously covered in your evaluation suite, and checkers should use deterministic string matching rather than relying on model self-evaluation.

Continuous Calibration with Human Annotation

Adversarial test cases catch known failures. But the capability gap keeps widening because new failure modes continuously emerge as users creatively explore the system. This requires a calibration feedback loop:

1. Sample production traffic (1–5% of requests, with PII scrubbed)
2. Route to human annotators using a minimal rubric: "Does this output satisfy the user's intent? Yes / No / Partial"
3. Compare human labels against evaluation predictions
4. Add inconsistent samples as new adversarial test cases

Here's a key insight: your evaluation suite is itself a model. It has precision and recall relative to human judgment. Track both. When recall drops — meaning the evaluation passes outputs that humans judge as failures — you have a blind spot, and those samples should be added to your adversarial set. Treating "partial" as a failure in the calibration code is a conservative but sound engineering tradeoff.

On treating the evaluation suite as a classifier with measurable precision and recall: In a binary classification framework: Precision = the proportion of outputs the evaluation marks "pass" that users are actually satisfied with; Recall = the proportion of outputs users are actually satisfied with that the evaluation correctly identifies. The more dangerous failure mode in practice is recall dropping — the evaluation lets through outputs users are actually unhappy with. These failures become invisible in metrics until a support ticket explosion makes them visible. Precision dropping is the cost in the other direction: an overly strict evaluation kills capabilities that could have shipped, causing the dev team to lose trust in it and start routing around it. Both need to be tracked, and ideally computed separately across different severity tiers rather than hidden behind a single aggregated number.

Measuring Real User Experience

Aggregate metrics hide the long tail. A system with 92% pass rate where the 8% failures are catastrophic is far worse than a system with 70% pass rate that degrades gracefully. What you should track is the severity of failures, not just the pass rate.

The approach is severity-weighted scoring: assign a weight to each failure category — prompt injection failures are security incidents (weight 5); JSON formatting errors break downstream parsing (weight 3); wrong multi-hop reasoning answers are noticeable to users (weight 2). The final score is computed as a weighted average, not a simple mean.

This design forces the team to prioritize what actually matters. A security-related injection failure with weight 5 should count for far more than a minor formatting glitch with weight 1. It makes "92%" have nowhere to hide — if the missing 8% is concentrated in high-weight categories, the weighted score will mercilessly expose the risk.

Automating the Feedback Loop

The endgame of this whole approach is wiring calibration output directly into the test suite. When annotators flag a new failure pattern, a script automatically generates an adversarial test case template. The process runs weekly via cron or GitHub Actions: pulling production logs and annotation data from S3, outputting date-stamped test files.

Auto-generated files clearly document their origin and pattern — for example, "Human annotators flagged 3 failures in 50 samples, pattern: model ignores 'CSV only' instruction when input contains commas" — along with a corresponding binary checker. After human review and merge, new cases run in CI on every PR.

This way, the evaluation suite evolves alongside the product rather than slowly rotting. It gets smarter every week with almost no manual maintenance cost.

Key Takeaways

• Golden datasets only measure average performance on clean inputs — they miss constraint violations, injection attacks, and reasoning failures, which are exactly what users actually encounter
• Build adversarial test cases from incident logs, turning every production failure into a binary test that blocks regressions
• Run a weekly calibration loop: sample production traffic, collect human labels, measure the evaluation's precision and recall against human judgment, and convert inconsistencies into new test cases
• Weight failures by severity so a 92% pass rate can't hide a 5% safety-critical failure rate
• Automate the entire loop so the evaluation suite grows on its own every week without manual upkeep

The methodology in this article draws on Karpathy's discussion of "the widening LLM capability consensus gap," supplementing it with concrete adversarial evaluation patterns, a runnable calibration loop, and a severity-weighted scoring system that wasn't covered in the original thread.