The Trap Behind a 97% Pass Rate: How Aggregate Metrics Hide Critical Failures

When a 97% Pass Rate Becomes a Lie

An evaluation suite with 1,800 test cases had been holding steady at a 97% aggregate pass rate. The dashboard was all green, and the team felt reassured — until they ran a cohort slice on their multilingual forms.

The result was sobering: the newly isolated 64-row multilingual cohort had a pass rate of just 61%. Yet the aggregate chart remained perfectly calm, while a human review queue for one particular language was quietly filling up with flagged cases. In other words, that top-line number had given almost no warning at all.

This real-world case from Reddit exposes a long-underestimated problem in AI evaluation systems: aggregate metrics naturally dilute concentrated failure patterns.

The Root Cause: Data Leakage and the Random-Split Trap

After digging deeper, the team traced the problem to their data split — specifically, data leakage.

The random assignment had placed nearly duplicate form templates on both sides of the train/test boundary. This made those "familiar structures" trivially easy for the model — what it saw at test time was essentially a variant of what it had seen during training.

This leakage introduced a subtle side effect: as the multilingual cohort's true performance began to degrade, those duplicated easy cases kept passing reliably, acting like ballast that kept the overall score almost completely stable.

Two compounding problems were at play:

• Data leakage stripped a portion of test cases of their evaluative value, artificially inflating the baseline
• Aggregate averaging spread concentrated failures across 1,700+ easy cases, making them effectively invisible

A deterministic scorer can reliably catch field format errors — but when you average that result across an entire suite, it can no longer tell you where the errors are concentrated.

Data Leakage in machine learning evaluation occurs when information from the test set has already been "seen" by the model during training, causing inflated evaluation results. In form extraction scenarios, this leakage often appears as template duplication: forms from the same company may share nearly identical field structures, and random splitting scatters these near-copies across both sides of the train/test boundary. The model is effectively recognizing rather than generalizing. The correct approach is to use a group or stratified split by source entity or template ID, ensuring all variants of a given template appear on only one side. This mirrors the logic used in traditional ML to prevent future data leakage with time-series data — the boundary must reflect a real decision boundary, not a statistical random cut.

Why Aggregate Metrics Systematically Mislead

The value of this case lies in the fact that it isn't an isolated bug — it's a structural flaw at the level of evaluation methodology.

Consider: 1,700 easy cases + 100 hard multilingual cases. Even if half of the hard cases fail, the aggregate pass rate might only drop from 97% to around 93% — a change easily dismissed as normal variance. But for the actual users of that language, the experience has completely broken down.

Averages obscure distributions — this is statistical common sense, yet it keeps catching engineering teams off guard. When your product serves multiple heterogeneous user groups (different languages, form types, regions), a single global threshold will almost inevitably result in the majority drowning out the minority.

The real danger isn't a low score. It's a score high enough to make you stop asking questions.

This phenomenon has a classic statistical counterpart: Simpson's Paradox, where a trend that holds within subgroups can disappear or even reverse when the data is aggregated. The "aggregation trap" in AI evaluation is its engineering variant: when subgroup sizes are highly uneven, the large group's high scores mathematically suppress the small group's low scores, making overall metrics look healthy while the affected group exerts almost no pull on the aggregate. This is especially dangerous in systems serving multilingual or multi-regional users — low-frequency languages tend to be exactly the ones with the thinnest data coverage, a double disadvantage that renders them collectively invisible on the dashboard.

The Fix: From Global Thresholds to Cohort-Level Quality Gates

The approach proposed by the original poster is worth borrowing for any team doing AI evaluation: pull critical slices out separately and stop letting them disappear into the overall score.

Specific steps include:

Maintain Critical Slices as Independent Regression Datasets

The author considered using tools like Braintrust to maintain the multilingual slice as a standalone regression dataset and set a dedicated quality gate for it in the CI pipeline. This way, if the slice degrades, CI blocks it immediately — rather than waiting for a human review queue to overflow before anyone notices.

Quality Gates are automated blocking mechanisms in CI/CD pipelines: when a metric falls below a preset threshold, the pipeline automatically fails and prevents the change from being merged or deployed. Applying this to AI evaluation means shifting the judgment of "is the model degrading?" from periodic manual review to a mandatory checkpoint that runs on every commit. LLM evaluation platforms such as Braintrust and LangSmith both support integrating evaluation datasets with CI hooks, triggering independent evaluation jobs on specific slices and returning a pass/fail signal. This is philosophically aligned with unit test coverage thresholds in software engineering — with one key difference: AI system "tests" are inherently probabilistic, so threshold definitions must additionally account for statistical confidence.

Examine Per-Case Experiment Diffs

Whenever modifying an extraction path, review per-case experiment diffs rather than just watching aggregate scores shift. Case-by-case comparison makes regressions concrete — "which case broke" rather than "the overall score dropped a few points."

Be Wary of Statistical Noise in Small Samples

The author also honestly raised an unresolved challenge: when a slice is small enough (say, only 64 rows), a handful of cases can swing the percentage dramatically — yet the slice is important enough that the overall score simply can't be trusted. How do you define a reasonable cohort-level threshold on a small-sample, high-stakes slice?

This remains an open question. Possible directions include: using absolute failure counts rather than percentages as the threshold for small slices, introducing confidence intervals, or setting differentiated pass criteria for different cohorts. But there's no silver bullet that simultaneously solves small sample size and high stakes.

When setting percentage thresholds for small cohort slices, confidence intervals are often too wide to be meaningful. Take 64 samples: if you observe a 61% pass rate, the 95% confidence interval (Wilson interval) spans roughly 48% to 73% — a swing of over 12 percentage points in either direction. A more robust alternative is setting an absolute failure count cap: for example, specifying that this slice must not exceed 10 failed cases, with any excess triggering the gate. This approach is insensitive to sample size and intuitively maps to the business semantics of "how many real users can we afford to impact?" Another direction is Bayesian methods: using historical failure rates as a prior, updating the posterior with new observations, and thereby conservatively estimating true risk when samples are scarce — avoiding false alarms or missed signals caused by random fluctuations in a handful of cases.

Key Takeaways for Evaluation Systems

For teams building evaluation pipelines for LLM or extraction-based systems, this case offers several directly actionable lessons:

First, never rely solely on aggregate metrics. Aggregate pass rates are useful for trend monitoring, but should never be the sole quality gating mechanism. At minimum, slice by key dimensions: language, use case type, and data source.

Second, audit your data splits for leakage early. Random splitting is dangerous when dealing with highly structured data that contains template duplication. Use group splits by template or source to keep training and test boundaries clean.

Third, set independent quality gates for high-value slices. Instead of chasing a single pretty global number, build separate guardrails for each critical cohort — so problems get caught where they actually concentrate.

A 97% pass rate that buys you blindness to a 61% failure slice isn't just worthless — it's actively harmful. Because it gives your team permission to feel at ease precisely when they should be on alert.