A Practical Guide to Building and Maintaining AI Eval Sets: Making LLM Application Quality Quantifiable

In application development powered by Large Language Models (LLMs), a critical yet often overlooked aspect is building and maintaining evaluation sets (eval sets). Many teams are eager to ship features quickly in the early stages of a project but lack a systematic evaluation mechanism to measure the quality of model outputs. As business requirements evolve and models iterate, teams without maintainable eval sets often find themselves in a "fix one thing, break three others" predicament. This article explores how to build a truly sustainable and maintainable eval set.

Why AI Eval Sets Matter So Much

An eval set is the "regression test suite" for AI applications. In traditional software engineering, unit tests and integration tests ensure that code changes don't introduce unexpected regressions. In LLM applications, however, model outputs are non-deterministic—a single prompt adjustment, a model version upgrade, or even a minor tweak to the temperature parameter can cause significant fluctuations in output quality.

This non-determinism represents a fundamental difference from traditional software testing. Traditional tests operate on a deterministic assumption—the same input must produce the same output. But LLM generation is based on probabilistic sampling; even with identical prompts and parameters, different calls may produce outputs that are semantically equivalent but textually different. This randomness stems from the sampling strategies used during model inference (such as top-p nucleus sampling and top-k sampling) and the degree of randomness controlled by the temperature parameter. The higher the temperature, the flatter the probability distribution and the greater the output diversity. Even setting the temperature to 0 in pursuit of deterministic output may still produce minor output differences in some API implementations due to floating-point precision variations and GPU batch processing optimization strategies. This means traditional assertEqual-style exact assertions are almost entirely ineffective for LLM testing, and evaluation must shift toward semantic-level quality judgments.

Without a stable evaluation baseline, developers can only rely on intuition to judge whether "this change made things better or worse." Such subjective judgment is extremely dangerous in production environments at scale. The core value of eval sets lies in transforming quality judgment from subjective experience into quantifiable, reproducible metrics.

Three Core Functions of Eval Sets

First, baseline anchoring. When you introduce a new model or modify a prompt, the eval set can immediately tell you whether performance has improved or degraded. Second, regression protection. It prevents you from inadvertently breaking scenarios that already work well while optimizing for a specific class of problems. Third, decision support. When you need to make trade-offs between cost, latency, and quality, quantified evaluation results provide objective criteria for those decisions.

It's worth emphasizing that eval sets play an irreplaceable role in detecting Performance Drift. Performance drift refers to the gradual degradation of model output quality over time, which is particularly common in LLM applications. Drift can originate from multiple levels: silent updates by model providers (e.g., OpenAI periodically updating their model snapshots without changing API endpoint names), shifts in contextual data distribution (user behavior patterns evolving with seasons or product lifecycle stages), and quality fluctuations in retrieval results caused by knowledge base content updates in RAG (Retrieval-Augmented Generation) systems. Since drift is typically gradual, it's difficult to detect through single observations—only through continuous monitoring via eval sets and historical trend comparisons can it be effectively identified. This is why evaluation cannot be a one-time activity but must be embedded into routine continuous operations.

Key Principles for Building Maintainable Eval Sets

Building an eval set isn't hard; making it maintainable is. A bloated, outdated, noise-filled eval set is worse than having no eval set at all, because it gives misleading signals.

Start from Real User Data, Not Imagination

The most valuable evaluation samples come from real user interaction data, not "ideal cases" that developers conjure up from thin air. Real data captures the edge cases, ambiguous expressions, and unexpected inputs that occur in actual usage. It's recommended to establish a mechanism for regularly sampling representative cases from production logs, especially those where the model performed poorly or users expressed dissatisfaction.

Sampling evaluation cases from production logs requires following statistical principles to ensure representativeness. Pure random sampling may lead to over-representation of high-frequency scenarios while ignoring edge cases—for example, if 90% of user queries are simple Q&A, a randomly sampled eval set will also be flooded with simple Q&A, failing to effectively test the model's performance in complex reasoning scenarios. A more effective strategy is stratified sampling: first classify user queries by intent type or cluster them via embeddings, then sample proportionally or equally from each category. Adversarial sampling should also be introduced—specifically collecting cases where the model has low confidence, users provide negative feedback (such as clicking a "not helpful" button), or fallback strategies are triggered. Although these "hard samples" may represent a small fraction of overall traffic, they are critical for evaluating model robustness and often expose the model's most vulnerable aspects.

Keep the Eval Set Lean and Focused

Many teams fall into the "more is better" trap, continuously piling samples into the eval set. However, an eval set containing thousands of redundant, duplicate samples is not only expensive to run but also dilutes critical signals. The ideal approach is to ensure each evaluation sample has a clear testing intent—what specific capability or failure mode is it testing?

Regularly reviewing and cleaning samples that are no longer relevant or highly redundant is an important part of maintenance work. The eval set should evolve alongside the product, not just grow indefinitely. A practical heuristic: if a particular evaluation sample has never failed in the past N evaluations, and the capability dimension it covers is already adequately covered by other samples, it's a candidate for redundancy cleanup.

Organize Eval Cases in Layers

Managing the eval set in layers by capability dimensions or scenario types significantly improves maintainability. For example, samples can be categorized into "core functionality," "edge cases," "safety and compliance," "format adherence," and so on. This layered structure allows you to quickly pinpoint which specific category of capability has degraded when analyzing results.

Layered organization also brings an important operational advantage: you can set different run frequencies and pass criteria for different tiers. For example, the "safety and compliance" category might require a 100% pass rate and must run on every change, while the "format adherence" category might allow a 95% pass rate and only run in daily builds. This differentiated strategy achieves a balance between evaluation comprehensiveness and runtime efficiency.

Choosing and Comparing LLM Evaluation Methods

The value of an eval set is highly dependent on the reliability of the evaluation method. Current mainstream evaluation approaches fall roughly into three categories.

Exact Match and Rule-Based Validation

For tasks with clear correct answers (such as classification, structured data extraction), exact matching or rule-based validation is the most reliable and cost-effective approach. These evaluation results are stable and reproducible and should be prioritized. Common implementations include: regex matching, JSON Schema validation, exact value comparison of key fields, and AST (Abstract Syntax Tree)-based validation of code generation results. The limitation of these methods is that they can only verify the formal correctness of outputs, not semantic quality.

LLM-as-Judge: Model as Evaluator

For open-ended generation tasks, another LLM is often needed to judge output quality. This approach is flexible but introduces new uncertainty—the judge model itself can make errors or exhibit biases. When using this method, it's essential to calibrate the judge's scoring criteria and regularly validate the judge's reliability with human-annotated results.

The theoretical basis of LLM-as-Judge is that a stronger model's quality judgments of weaker model outputs exhibit high consistency with human evaluations. Research by Zheng et al. in 2023 ("Judging LLM-as-a-Judge") showed that GPT-4 as a judge can achieve a Spearman correlation coefficient above 0.8 with human expert evaluations. However, this method has several well-documented systematic biases: position bias (tendency to prefer the option listed first in pairwise comparisons), verbosity bias (tendency to score longer, more detailed answers higher, even when the additional content adds no informational value), and self-enhancement bias (tendency to prefer outputs similar to its own generation style). Mitigation strategies include: multiple judgments averaged to reduce randomness, randomizing option presentation order to eliminate position effects, using structured rubrics that explicitly define the specific criteria and examples for each score level, and cross-validation using multiple different judge models.

Human Evaluation as the Quality Benchmark

Human evaluation is the gold standard for quality, but it's expensive, slow, and difficult to scale. A reasonable strategy is to use human evaluation for calibrating automated evaluation methods and handling high-value cases that automated approaches cannot adequately cover.

In practice, human evaluation typically employs Inter-Annotator Agreement (IAA) metrics to ensure the reliability of the evaluation quality itself. Common measures include Cohen's Kappa and Krippendorff's Alpha. If agreement between annotators is low, it usually means the evaluation criteria are not clearly defined and the rubric needs refinement. A validated workflow is: first have multiple annotators independently judge a small set of samples, calculate agreement, iteratively refine the judgment criteria until an acceptable level of agreement is reached (typically Kappa > 0.7), and then proceed with large-scale annotation.

Integrating Evaluation into the CI/CD Development Workflow

Eval sets only deliver value when they are used continuously. Integrating the evaluation process into the CI/CD pipeline so that every prompt or model change automatically triggers evaluation is a key step toward achieving maintainability.

At the engineering implementation level, integrating LLM evaluation into CI/CD workflows typically requires addressing several key challenges. First is cost control—running the full eval set on every code commit can generate substantial API call costs (especially when using high-end models like GPT-4 as judges). A common approach is a tiered evaluation strategy: run only a core subset during the Pull Request stage (similar to a smoke test covering critical paths), run the full evaluation suite when merging to the main branch, and run deep evaluations including LLM-as-Judge via scheduled daily tasks. Second is latency—LLM evaluation involves numerous API calls and may take minutes to tens of minutes to complete, requiring asynchronous execution mechanisms that deliver evaluation results as non-blocking check reports rather than hard gates (or set hard gates only for critical categories). Finally is result interpretation—clear pass/fail thresholds must be established (e.g., core metrics must not fall below 95% of the baseline), along with alerting mechanisms and automated regression reports. The current ecosystem includes several dedicated LLM evaluation platforms that support such integration, such as Braintrust, Promptfoo, LangSmith, and Ragas, which provide version comparison, regression detection, visualization dashboards, and GitHub/GitLab integration, significantly lowering the engineering barrier.

When evaluation becomes a routine part of development rather than an afterthought, teams can truly build confidence in their AI application quality. Additionally, establishing historical tracking of evaluation results helps teams observe long-term quality trends and detect gradual performance drift in a timely manner. This philosophy of continuous monitoring is consistent with observability in traditional DevOps—just as we wouldn't run production services without monitoring, we shouldn't run LLM applications without continuous evaluation.

Conclusion

Building a maintainable eval set essentially means bringing the rigor of software engineering into the uncertainty-filled world of LLM application development. The core principles are: start from real data, keep it lean and focused, organize in layers, choose appropriate evaluation methods, and integrate evaluation into the daily development workflow.

For any team that takes AI product quality seriously, the eval set should not be viewed as a one-time task but rather as core infrastructure requiring long-term investment and continuous maintenance. It determines whether your AI application can maintain stable, reliable quality performance through rapid iteration cycles.