Metrics First, Dashboard Second: A Guide to Choosing Metrics for LLM Evaluation

Many teams invest heavily in building polished LLM evaluation dashboards, only to discover they still know nothing about whether the product actually works. The problem usually lies in the order of operations: they pick a metric that "looks authoritative," then pile charts around it, and end up with production behavior and dashboard curves telling two different stories. The right approach is the reverse—first clarify your feature type (classifier, ranker, generator, or retrieval-based QA), then choose evaluation metrics that match that type.

Classifier-Type Features: F1, Precision/Recall, and AUC

If your LLM feature ultimately outputs a yes/no judgment—such as whether content is toxic, whether a case needs escalation, or intent A versus intent B—then even if the model reasons internally in natural language, it is fundamentally a classification problem and should be evaluated with classic classification metrics.

Why accuracy deceives

In label-imbalanced settings, accuracy is highly misleading. A model that always predicts the majority class can score high on accuracy while catching none of the cases you actually care about. The F1 score is the harmonic mean of precision and recall; weakness on either side drags F1 down, so a model cannot game it by "maximizing precision while missing all positives" or "predicting everything as positive to inflate recall."

The author's approach: when you need to judge which failure mode hurts users more, look at precision and recall separately. F1 weights both equally, but your error costs are not necessarily equal.

Class imbalance refers to a large disparity in the number of samples across classes—for example, in abuse detection, 99% of samples are normal and only 1% are abusive. In this case, accuracy skews toward the majority class: a model that classifies all samples as "normal" can still achieve 99% accuracy while completely missing all the abuse cases that actually need attention. A common metric for measuring the impact of imbalance is prevalence, the proportion of positives in the data; the lower the prevalence, the more easily accuracy is inflated, while precision, recall, and F1 are more sensitive to performance on the minority class. Therefore, in label-imbalanced settings, these metrics should be prioritized to evaluate the model's ability to identify the minority class.

The applicable boundaries of AUC

When the model outputs a score and the decision threshold has not yet been fixed, AUC-ROC is suitable for comparing different models on the same dataset. It measures the ability to rank positives above negatives as the threshold slides. An AUC near 0.5 means near-random performance on that dataset; near 1.0 means strong discrimination.

But in rare-positive scenarios (fraud, abuse, defect detection), AUC can look beautiful while most actual positive predictions are still wrong—because on a huge negative pool, even a low false positive rate produces many false alarms. When positives are scarce, report precision and recall, or report F1 at the threshold you plan to deploy, alongside AUC.

AUC-ROC measures the model's ability to rank positives above negatives, equivalent to the probability that a randomly drawn positive scores higher than a randomly drawn negative. It only cares about ranking, not any specific threshold, so it is suitable for comparing models before a threshold is fixed. Precision is the proportion of predicted positives that are truly positive; recall is the proportion of actual positives that are correctly predicted. In rare-positive scenarios, even if the false positive rate (FPR) is low, the absolute number of false positives can still far exceed true positives because the negative pool is enormous, leading to very low precision. Therefore, a high AUC does not mean few actual false alarms; it must be combined with precision, recall, or F1 at a business-viable threshold for a comprehensive judgment.

Generator-Type Features: What Perplexity Can and Cannot Do

Autoregressive models assign probabilities to the next token, and perplexity aggregates these probabilities into a single number: the lower it is, the less surprised the model is by held-out text. During training, or when comparing models on the same unseen text under the same tokenizer, perplexity is a useful signal. But once the tokenizer changes, or test data leaks into the training set, the comparison breaks down.

More importantly, perplexity measures text prediction ability, not success on your task. Lower perplexity does not guarantee better production behavior. A model can assign high probability to fluent continuations while still hallucinating, ignoring instructions, or failing your grading rubric. The author's position is clear: perplexity is a development-stage signal for comparing like models, not evidence you can deliver to users for chat, Copilot, or agent features.

What generators really need is task-level evaluation: rubrics, human review, model judges where appropriate, and checks against user requests. Only end-to-end measurement of the application tells you whether the feature is usable.

Perplexity is a measure of how "surprised" a language model is by a piece of text, mathematically the exponent of the average negative log-likelihood the model assigns to each token. Lower perplexity means the model predicts the text more accurately. But perplexity depends heavily on the tokenizer: different tokenizers split tokens at different granularities, so the same text has different token counts and perplexity values cannot be compared directly. Moreover, perplexity only reflects the model's statistical prediction ability over text; it says nothing about factual correctness, instruction following, or safety. Therefore, it is suitable for comparing training progress across models with the same tokenizer and the same held-out set, but cannot serve as evidence of task-level success or production quality.

RAG and Citation-Type Features: Faithfulness, Not a Single Accuracy Number

Retrieval-augmented generation and cited answers raise a different question: did the model stay grounded in the retrieved context, or did it invent details? Faithfulness (sometimes called groundedness) checks whether the claims in an answer are supported by the provided sources, targeting hallucinations of the "added content not present in the context" variety.

The key distinction: faithfulness and answer correctness are two different things. An answer can be perfectly faithful to a wrong source document. So it needs to be paired with context relevance (was the right material retrieved?) and answer relevance (did the model answer the question?), rather than serving as a single headline KPI.

Alex Ratner of Snorkel AI put it bluntly on Episode 57 of Chain of Thought: to get models to do different things, it's not about "brain surgery" on neurons or reinventing the algorithm architecture—it's about getting data. The same idea applies to evaluation data: choose the right measurement instrument for your feature, then invest in labels and rubrics so gaps surface before users find them.

Faithfulness, sometimes called groundedness, measures whether every claim in a generated answer can be traced to the retrieved context. It focuses on whether the model "fabricated" content outside the context. Context relevance measures whether the retrieved documents are relevant to the user's question; answer relevance measures whether the generated answer directly addresses the question. Together these three form the triangle of RAG evaluation: faithfulness prevents hallucination, context relevance ensures retrieval quality, and answer relevance ensures the answer stays on point. An answer can be faithful to a wrong source document (faithful but wrong), so all three must be combined rather than relying on a single metric.

Hands-On: Running F1 and AUC on a Labeled Held-Out Set

For binary classification features, start with a labeled held-out set that reflects the skew of your production distribution. On a validation slice, choose a threshold based on the false positive and false negative costs you can accept, then report F1 at that threshold—that is the story stakeholders can feel. Also keep AUC on scored samples for comparing models before locking in a cutoff.

Here is a minimal Python illustration (not production code):

# Illustration: binary labels, thresholded predictions, and scores on a held-out set
from sklearn.metrics import f1_score, roc_auc_score, precision_recall_fscore_support

y_true = [0, 1, 1, 0, 1]
y_score = [0.2, 0.9, 0.7, 0.1, 0.6]
threshold = 0.5
y_pred = [1 if s >= threshold else 0 for s in y_score]

f1 = f1_score(y_true, y_pred)
auc = roc_auc_score(y_true, y_score)
p, r, _, _ = precision_recall_fscore_support(y_true, y_pred, average="binary")
print(f"F1={f1:.3f}  AUC={auc:.3f}  precision={p:.3f}  recall={r:.3f}")

For multiclass routing, use per-class F1 and choose an averaging method you can defend with error costs.

Metric Selection Checklist

• Classifiers with label imbalance: report F1 at the production threshold, plus precision and recall; use AUC on the same data to compare models before fixing a cutoff.
• Rare positives: do not treat AUC as sufficient; check precision at a viable threshold.
• Generator features: use perplexity only for comparable models on held-out text, then do task-level evaluation of production behavior.
• RAG or citation features: faithfulness alongside context relevance and answer relevance; a faithful but wrong answer will still disappoint users.
• Draw the metric map first, then build the dashboard.

The core idea can be summed up in one sentence: measurement data is also data—choose the right instrument, then invest in labels and rubrics that reveal real gaps. Metrics first, dashboard second, so evaluation does not become mere decoration.

One-line Summary

Define the feature type before choosing evaluation metrics, so the dashboard does not become mere decoration.

Paragraph Summary

The article points out that many teams build LLM evaluation dashboards in the wrong order: picking metrics first and then piling on charts, which leaves production behavior disconnected from the dashboard. The right approach is to first clarify the feature type (classifier, generator, RAG QA), then choose matching metrics. Classifiers should focus on F1, precision/recall, and AUC, especially noting how misleading accuracy is under label imbalance; generator perplexity is only suitable for comparing text prediction across comparable models and cannot prove task success; RAG features need faithfulness as the core, paired with context relevance and answer relevance. The article also provides hands-on examples and a metric selection checklist, emphasizing "draw the metric map first, then build the dashboard."