ROC-AUC 0.95 but Flooded with False Positives: Why Rare Events Need PR-AUC

While monitoring social media during a wildfire, your classifier achieves a ROC-AUC of 0.95—which looks nearly perfect. But when you actually open the alert list, the vast majority of flagged posts turn out to be irrelevant. A high score coexisting with high false positives doesn't mean the model is broken; it means a single metric is masking the true cost of the rare positive class problem.

Based on a reproducible Python tutorial, this article uses a deliberately constructed synthetic dataset to break down what ROC-AUC and Precision-Recall metrics each answer, and why in scenarios with a 1% positive class ratio, you must look at both.

Rare Events: 99% Accuracy Can Be Meaningless

Imagine 10,000 social media posts, of which only 100 contain events that truly require attention—evacuation notices, road closures, emerging hazards. The positive class accounts for 1%—a classic case of extreme class imbalance.

Such distributions are not uncommon: fraudulent transactions, security breaches, equipment failures, crisis event reports—the positive class is always the minority. The original article offers a sharp reminder: a model that classifies every post as negative achieves 99% accuracy while finding not a single positive.

Metrics are only useful when they reflect the problem you're actually trying to solve. Accuracy carries almost no information here because it is dominated by the 9,900 negative samples.

What ROC-AUC and PR-AUC Measure

Both metrics are based on the confusion matrix, but they use different denominators, leading to vastly different sensitivities to the rare positive class.

ROC-AUC: Ranking Ability for the Positive Class

The ROC curve plots the false positive rate (FPR = FP / (FP + TN)) on the x-axis and the true positive rate (TPR = TP / (TP + FN)) on the y-axis. ROC-AUC summarizes the model's ability to rank positives above negatives across all thresholds. 1.0 is perfect ranking, 0.5 is close to random.

The problem lies in the denominator of FPR: it uses all negative samples. When there are thousands of negatives, a seemingly small false positive rate, multiplied by the huge denominator, still produces a large number of false positives. An FPR of 0.05 on 9,900 negatives translates to 495 false positives.

PR-AUC: The Trustworthiness of Positive Predictions

The Precision-Recall curve asks a different set of questions. Precision (Precision = TP / (TP + FP)) asks: of the posts flagged, how many are actually relevant? Recall (Recall = TP / (TP + FN)) asks: of the truly relevant posts, how many were retrieved?

In scikit-learn, Average Precision (AP) is typically used to summarize the PR curve. The original article specifically notes that AP is a common summary of the PR curve but is not equivalent to directly computing the trapezoidal area under the curve.

For imbalanced data, precision-recall analysis often reveals the real consequences of false positives more effectively than ROC analysis alone (Saito & Rehmsmeier, 2015).

A Deliberately Simple Reproducible Experiment

The original article does not train a complex model but constructs synthetic data with known scores, aiming to isolate the behavior of the metrics themselves. The classifier's score distribution is as follows: 495 negative posts score as high as 0.90, ranking above all positives; 80 positive posts score 0.80, and 20 positives score 0.60; the remaining 9,405 negatives score 0.10.

Using scikit-learn's roc_auc_score and average_precision_score, computed at a threshold of 0.5, the output is worth examining line by line:

ROC-AUC: 0.9500
Average Precision: 0.1449
Precision at threshold 0.5: 0.1681
Recall at threshold 0.5: 1.0000
Accuracy at threshold 0.5: 0.9505

True positives: 100
False positives: 495
True negatives: 9405
False negatives: 0

ROC-AUC of 0.95 suggests strong ranking ability. But AP is only about 0.145, and at the 0.5 threshold, precision is under 17%—of the 595 flagged posts, only 100 are truly relevant. Recall of 100% means no event was missed, at the cost of 495 false positives.

If this were a crisis response team, manually reviewing 595 posts to confirm 100 valid pieces of information represents considerable workload when the system runs continuously.

The Difference Between Average Precision and the Area Under the PR Curve

Average Precision (AP) is the default summary method for average_precision_score in scikit-learn. It computes the weighted average of precision over recall, with weights being each recall increment. Specifically, AP equals Σ (R_n - R_{n-1}) * P_n, where P_n and R_n are the precision and recall at the n-th threshold. In contrast, directly computing the trapezoidal area under the PR curve (AUC-PR) uses linear interpolation, and the two differ when the curve is non-monotonic. AP is more commonly used because it is more sensitive to sudden drops in precision and is widely accepted as the standard metric for imbalanced classification. Understanding this helps correctly interpret the 0.1449 in the experiment output and avoid mistaking it for a simple geometric area.

The Difference Exposed by the Two Curves

Plotting the ROC curve and the PR curve side by side, the difference is immediately clear. The ROC curve shows strong overall ranking performance; the PR curve visually captures the precision collapse caused by 495 negative posts scoring higher than all positives.

The PR plot also has an easily overlooked reference line: the positive prevalence. In this example, it is 1%. The expected AP of a random ranking model is approximately equal to the prevalence, i.e., 0.01. In other words, while AP 0.145 looks low, it is still clearly better than random—just far less optimistic than ROC-AUC 0.95 suggests.

The original article also offers a necessary caveat: this dataset uses deliberately constructed score groupings; real models typically have different score distributions and different curve shapes.

Is PR-AUC Always More Important? Not Necessarily

This is a conclusion that is easy to oversimplify. The original article explicitly states: not necessarily.

ROC-AUC remains valuable for comparing ranking behavior across thresholds, especially when both positive and negative classes matter. The advantage of PR metrics is concentrated in scenarios where the positive class is rare and the quality of positive predictions directly impacts operations.

The choice depends on the cost of errors. The original article provides two contrasting scenarios:

• High cost of missed detections (early crisis detection): Tending to lower the threshold to capture as many potentially important posts as possible increases recall, along with false positives.
• Limited manual review capacity (a small team can only review a few posts): Precision and PR behavior become critical; a model flagging too many irrelevant posts directly overwhelms reviewers.

The threshold should reflect the task, available resources, and the relative costs of the two types of errors. The AUC value itself will not choose the threshold for you.

Positive Prevalence and the Random Baseline

Positive prevalence is the proportion of positive samples in the dataset, which is 1% in this example. For a randomly ranking classifier, the expected Average Precision is approximately equal to the prevalence, because without any information, the proportion of positives among flagged samples is simply the overall positive proportion. Therefore, while AP 0.145 is not high in absolute terms, it still represents a significant improvement over a random baseline of 0.01. This reminds us that when evaluating PR metrics, we must interpret them in conjunction with prevalence: a model with AP 0.2 may perform well in a scenario with prevalence 0.01 but poorly in one with prevalence 0.5. Prevalence also explains why the PR curve is more sensitive than the ROC curve when classes are extremely imbalanced—it directly reflects the purity of positive predictions.

Anomaly Detection and Fact-Checking: Two Responsibilities, Two Evaluation Approaches

The original author, in their master's thesis, researched social media anomaly detection and fact-checking in crisis management scenarios, and proposed an important design principle: anomalous content should be prioritized for analysis but should not be automatically treated as disinformation.

Anomaly detectors identify anomalous signals—suddenly clustering posts, spikes in message volume, content deviating from normal patterns. These are merely leads for further investigation, not evidence that a post is false. The system still needs to verify claims, assess available evidence, and handle uncertainty.

Therefore, evaluation should also be separate: anomaly detection tests "can relevant cases be ranked at the top," while fact-checking must be evaluated against appropriate reference evidence and labels. This distinction is not merely philosophical; it determines what the system measures, how outputs are interpreted, and which errors truly matter.

A Practical Checklist

The checklist from the original article can be used directly for routine model evaluation:

1. Check the class distribution first: Before interpreting any score, determine how rare the positive class actually is.
2. Don't report accuracy alone: Look at ROC-AUC, Average Precision, precision, recall, and the confusion matrix together.
3. Choose thresholds by consequences: Think clearly about the workload caused by false positives and the real harm caused by missed detections.
4. Validate on representative data: Performance on a controlled dataset does not automatically prove usefulness in the real environment.

The most essential point is: evaluate the model against the task it is truly meant to support. In AI-assisted decision-making, the goal is not to produce a pretty score, but to understand what that score proves and what remains uncertain.