Accuracy

In scenarios with extreme class imbalance, even an accuracy as high as 95% may indicate that the model is completely unable to identify the minority class.

When evaluating a judge, one should look at the confusion matrix rather than just accuracy; if 90% of the samples are good answers to begin with, a judge that passes everything can still show 90% accuracy.

Accuracy only checks whether the highest-probability option is correct, while NLL also measures the probability assigned to the correct option, so NLL can worsen while accuracy rises.

Most benchmarks use accuracy as the core metric, counting both wrong answers and refusals as incorrect and not distinguishing the difference in harm.

Under label imbalance, accuracy is highly misleading: a model that always predicts the majority class can score very high on accuracy while missing every positive example.

In multi-class tasks, accuracy becomes distorted when class samples are imbalanced; precision reflects the degree of false positives, recall reflects the degree of false negatives, and the F1 score is the harmonic mean of precision and recall.

Vector distance and retrieval thresholds are a trade-off between recall and accuracy: the looser the threshold (the larger the value), the more is recalled, but the higher the probability of irrelevant content being mixed in; tightening the threshold may miss answers that could have been reused.

Accuracy as a moderation metric has a structural flaw: it treats false positives and false negatives as equally erroneous, but in an e-commerce store the cost of wrongly blocking a customer is far higher than that of letting a bad review through, so one should focus on the absolute numbers of detections and false positives rather than on accuracy rankings.

In classification tasks with imbalanced classes, accuracy can be misleading; for example, if only 20 out of 1000 tickets are truly urgent, a model that predicts 'not urgent' for all of them can still achieve 98% accuracy.

Average accuracy masks distributional skew; a judge that gets all easy cases right and is overconfident on hard cases can post a high average score while failing at critical points.

No single metric deserves complete trust; accuracy and calibration should be used together, and it should be confirmed that evaluation questions are consistent with deployment scenarios.

An evaluation culture centered on single metrics such as accuracy and F1 may be measuring in the wrong direction when faced with inherently ambiguous tasks.

Most evaluation metrics such as precision and F1 cannot distinguish between a system that checked and found no relevant evidence and a system that did not check at all.

In a fraud detection task with a positive-to-negative sample ratio of 1:99, a model that predicts all samples as 'non-fraud' can still achieve 99% accuracy.