How to Ensure Quality of AI Features That Can't Be Unit Tested with Evals

Traditional unit tests break down in the face of AI features—run the same summary request twice and you get different but equally reasonable results. Teams often fall back on manually trying a few prompts and shipping, which is really a demo, not a test. This article introduces a systematic approach to measuring AI output quality using eval sets: building test sets from real inputs, writing down a definition of a "good answer," scoring through a three-layer mechanism of code checks + human review + model grading, and agreeing on non-negotiable baselines before launch.

Why Unit Tests Fail for AI Features

Every developer has muscle memory: feature done, write a test that feeds input and asserts output. Save succeeds, date parses correctly—these can all be locked down with expect(output).toBe(expected).

But AI features break this reflex. Ask the model to summarize the same lecture twice and you get different summaries, yet both make sense; ask the model to classify customer service messages and it's right 99 times, then confidently wrong on the 100th. There's nothing for the assertion to grab onto.

The danger isn't a failing test—it's the team simply giving up on testing. Someone manually picks a few prompts, runs them, reads the answers, thinks "looks fine," and considers it passed. That's a demo, not a test—because the person picking the inputs also decided what counts as passing.

The alternative to unit tests is called an eval set. It's not a platform; it's three things: a collection of real inputs, a written definition of what a "good answer" looks like for each input, and a recorded score after every change. The score itself matters less than the trend: you don't need to prove the feature is perfect, you just need to know whether Thursday's prompt change made things better or worse.

Use Real Inputs, Not the Inputs You'd Pick

Eval samples must come from actual workloads: real customer service messages, real documents, real call recordings. Made-up examples are always cleaner than reality, and a feature tuned on clean samples falls apart the moment it hits dirty data.

When you don't have production data, use the closest thing: old tickets, sample documents from early customers, pilot recordings—anything produced by people who didn't know a model would read it.

Then deliberately hunt for uncomfortable cases:

• Ambiguous requests: the correct behavior is to ask for clarification, not to guess.
• Out-of-scope requests: for example, a booking assistant asked for legal advice should politely decline or hand off to a human.
• Bad inputs: blurry scans, half-filled forms, background noise, mixed languages.
• Adversarial inputs: text that tries to make the system ignore its instructions—especially critical when the system can take actions.
• High-stakes cases: scenarios where a wrong answer costs money, loses trust, or creates safety issues.

A few dozen carefully chosen samples like these expose real problems better than thousands of synthetic examples nobody looks at. The eval set grows over time as production data comes in.

Writing Down the Definition of "Good" Is the Real Work

For each sample, someone who understands the business writes down what a good answer looks like. Sometimes it's precise: the invoice total is this number, the caller wants Tuesday at 3 PM. More often it's a short checklist: mentions the deadline, doesn't promise a refund, stays under one paragraph, hands off to a human when the customer is upset.

The value is mainly here, and it has nothing to do with code. Writing definitions forces the team to agree on what the feature is supposed to do. If two people want opposite behaviors from the same assistant, it's best to discover that while filling in row 12 of the spreadsheet—arguing in week two is free; arguing after launch is expensive.

Three Scoring Methods, All Necessary

Code checks. Whatever a function can verify, let a function verify: are the extracted fields correct, is the date valid, did the voice assistant call the booking tool with the right time slot, is the reply too long, does it leak a link it shouldn't share. These are fast and cheap, run on every change, and are the closest thing to old-fashioned unit tests—a good place to start.

Human review against a rubric. Some qualities only humans can judge: is the summary accurate, is the tone appropriate. The trick is making the rubric specific enough that two reviewers give the same output the same score. "Rate quality from 1 to 5" produces noise; "Does it state any fact not present in the source text" produces signal.

Model grading model output. Using a second model to score outputs against a rubric scales to places human effort can't reach, and it genuinely works. But it has its own consistent biases. I treat it as an assistant to human review: regularly validate its scores against human samples, and never let it be the sole judge in high-stakes scenarios.

Model grading model output (LLM-as-a-judge) in practice usually works like this: put the original input, the output being evaluated, and the rubric together into the judge model's prompt, and ask it to output a structured score or a pass/fail judgment. This is far cheaper than human review and can run in batches, but there are several known biases to watch for: judge models tend to give higher scores to longer answers (length bias), prefer outputs similar to their own generation style (self-preference bias), and produce position bias when scoring criteria are vague (the same pair of outputs flips conclusion when swapped in order). Mitigations include: listing explicit checkable items in the rubric rather than using a vague 1-5 scale, regularly sampling for human review and calculating the agreement rate between judge model and human scores, and keeping human final review for high-stakes scenarios.

A Test Harness Can Be Embarrassingly Small

People hear "evals" and think platform. The first version I recommend is a spreadsheet (input + expected checks) plus a script that runs everything, prints the score, and compares it against last time. Illustrative code (not from a customer repo):

type EvalCase = {
  id: string;
  input: string;
  checks: Array<(output: string) => boolean>;
  mustNotFail?: boolean; // e.g. never quote a price
};

async function runEvals(cases: EvalCase[], run: (input: string) => Promise<string>) {
  let passed = 0;
  const hardFailures: string[] = [];

  for (const c of cases) {
    const output = await run(c.input);
    const ok = c.checks.every((check) => check(output));
    if (ok) passed++;
    else if (c.mustNotFail) hardFailures.push(c.id);
  }

  const score = passed / cases.length;
  console.log(`score ${(score * 100).toFixed(1)}%  hard failures: ${hardFailures.length}`);
  return { score, hardFailures };
}

What's interesting isn't the loop—it's the mustNotFail flag, which leads to the mistake I've seen most often.

Set the Launch Bar Before You See the Score

Once teams see a score, it's easy to argue "this score is acceptable." 87% sounds good—especially after three weeks of work.

So agree on these with the person who owns the outcome, in advance:

• How many points on typical cases are needed to launch.
• Which failures are unacceptable at any rate: medical or legal advice, wrong prices, other customers' data. These are mustNotFail cases—one is enough to veto a release, no matter how high the average score.
• What happens when the system isn't sure. Usually the right design is to hand off to a human, turning a potential failure into a safe outcome.

The difference between mustNotFail cases and ordinary eval cases is semantic: ordinary cases contribute to the pass rate, while mustNotFail cases are hard constraints—any single failure vetoes the release, regardless of the average score. This design corresponds to the "safety invariant" idea in engineering: average quality can be traded off, but certain failure modes (leaking others' data, quoting wrong prices, offering medical or legal advice) are unacceptable at any rate. The point of agreeing on the bar in advance is to avoid the "score anchoring effect"—after seeing 87%, teams easily argue the number is acceptable, but if the bar written down before launch was 95%, then 87% is a clear signal of falling short, not a number that needs defending.

It Only Pays Off After Launch

AI features change differently from ordinary code, and most changes never show up in a diff review:

• Prompt edits: wording changed to fix one problem can silently break three others.
• Model updates: providers update and deprecate models; the same model name can behave differently month to month, and eventually you may be forced to switch.
• Data changes: if the feature answers from documents, reorganizing the documents changes the answers.
• Cost changes: switching to a cheaper model to control spend is reasonable—but only if you can prove quality didn't drop.

Run the eval set on every change like this, and a tense subjective judgment becomes a comparison. Put the old score and the new score side by side where the team can see them, and let regressions block changes like failing tests.

The eval need created by model updates is often underestimated. Provider updates to a model usually don't change the model name, yet behavior can drift month to month; and model deprecation often gives a limited migration window, forcing teams to switch under time pressure. Both cases mean the eval set must be re-runnable quickly without code changes, with historical scores kept for comparison. Treating every model version change, prompt edit, and data source adjustment as a "change event" that requires running evals is the organizational habit that makes an eval set actually work—not just the technical preparation.

Production Is the Real Source of Eval Sets

Pre-launch evals are an educated guess about traffic; production is the truth, and the eval set should keep learning from it:

• Log enough to investigate (within privacy rules): input, output, system decisions.
• Watch for failure signals: users asking the same question again, requesting a human, staff correcting results, thumbs-down.
• Turn every real failure into a new case, write down the correct answer, and make the same mistake get caught automatically next time.

Voice scenarios make this most obvious. Real calls have interruptions, changed minds, and requests the business doesn't handle; replaying these is part of checking changes. Phone calls are fast and have no undo button—which is why another related article of mine is called "Phone Calls Can't Be Reproduced."

It's the same outside voice. For an app that turns lecture recordings into summaries and outlines, the useful eval questions are specific: did the summary miss a key point, or add something the lecturer never said? These two checks catch problems better than any generic "quality score."

The Honest Cost

The first version of an eval set isn't a big project—it's a few days of work during development, most of it spent with business-savvy people writing down the definition of "good." What really costs is attention: someone must own the definition of "good" and review failure cases regularly. If that person is "nobody," the eval set goes stale and you're back to shipping on vibes.

What does your eval setup look like right now? I'm especially curious whether anyone trusts model grading in high-stakes scenarios, and how you validate it.