Human Review vs. Automated Evaluation: Who Should Score Your AI Agent?

AI Agent evaluation is a challenge every team eventually has to face. Once the prototype works, someone always asks: who checks these outputs? Many teams hope AI can handle it on its own, but reality often proves the opposite—teams that skip this step end up shipping an Agent no one can vouch for.

This article draws on production practices from teams at Netflix, Rhesis, and others to compare human expert evaluation and automated evaluation side by side, analyze their respective strengths and breaking points, and propose a practical hybrid evaluation loop.

Human Evaluation: Domain Knowledge That Automation Cannot Replace

Context exists only inside your organization. Generic metrics like helpfulness, toxicity, and faithfulness struggle to judge whether your travel Agent got your fare rules right. For example, it tells the user they can cancel for free up to 24 hours before departure, when the actual policy is 48 hours—the answer is fluent, polite, and backed by fare documentation, but it's wrong. Only the operations team knows.

Experts spot problems extremely fast. Hand ten conversation logs to a customer service operations colleague, and within seconds they'll find the one that used last season's baggage allowance. Hamel Husain recommends designating a primary domain expert as the single source of truth, making binary pass/fail judgments on each output with written commentary. The commentary matters as much as the judgment, because it's the raw material that later becomes metrics.

Evaluation criteria emerge from reading outputs, not from imagination. Shankar et al. call this phenomenon criteria drift: people often clarify their own criteria gradually during the scoring process, revising as they see more outputs. A scoring rubric written before anyone has read the conversation logs is just a first guess.

Breaking point: Experts are slow, expensive, and busy. Asking them to manually write 200 test cases isn't realistic, and re-reading every conversation each time you change a prompt is even more absurd.

Ye Yuan hit exactly this wall while building customer service bots across travel, healthcare, education, and legal services. Generic NLP metrics couldn't capture multi-turn conversation quality, so the team had to build their own scoring criteria, using the best human sales and support conversations as a benchmark. The first bottleneck was annotation capacity: training and calibrating annotators was both slow and expensive, and the gold-standard dataset remained too small.

What are helpfulness, toxicity, and faithfulness? These three metrics are the most common generic dimensions in current LLM evaluation. Helpfulness measures whether an answer truly solves the user's problem rather than just being formally complete; toxicity detects whether outputs contain offensive, discriminatory, or harmful content; faithfulness targets RAG-style systems, examining whether answers strictly follow retrieved context rather than being fabricated by the model. Their shared limitation: they're all domain-agnostic, making them nearly useless for business fact errors like "fare rules written as 24 hours."

Automated Evaluation: A Powerhouse for Scale and Adversarial Testing

Explore an Agent without any prerequisite information. Point an automated tester at an interface, and it can probe the domain, map the Agent's capability boundaries, and find where it refuses or escalates. No requirements document needed.

Run adversarial tests at scale. Jailbreaks, prompt injection, and multi-turn pressure campaigns all require scale and variation. Manually covering the OWASP LLM Top 10 and Agent application security checklists isn't realistic, and generating and executing these attacks is exactly where automation shines.

Continuously verify requirements after handoff. Every run, every model swap, every prompt change—all run in CI. Human reviewers are exhausted by the 40th conversation; metrics never tire.

Breaking point: The devil is in the details. LLM judges tend to be lenient, handle partially correct answers poorly, and are highly sensitive to the wording of the judge prompt (Thakur et al., "Judging the Judges"). They miss small errors that an expert would catch immediately—24 hours vs. 48 hours. More insidiously, agreement rates mask problems: if only 5% of outputs fail, a judge that passes everything has a 95% "agreement rate" with the expert while catching zero failures.

Ye Yuan saw the same problems in production: LLM judges struggled with pragmatics, implicature, and scenarios where users stay silent, and scores fluctuated noticeably between runs. Synthetic test conversations also lost the ambiguity and emotional/topic shifts present in real user traffic.

LLM-as-a-Judge and the agreement rate trap. Using one large model to judge another model's output is called LLM-as-a-Judge, the mainstream paradigm for automated evaluation. It's cheap and batch-runnable, but has known biases: it favors long answers and high scores, is sensitive to wording, and struggles with "partially correct." The "agreement rate paradox" proposed by Thakur et al. is especially worth noting—assuming the true failure rate is only 5%, a judge that always passes still has 95% agreement with the expert, but its recall is zero. Therefore, evaluating a judge requires looking at precision, recall, and the confusion matrix together, not just agreement rate.

How the Two Complement Each Other

Experts excel at identifying problems but are weak at producing scale; automation is exactly the opposite. So the truly useful question isn't "which one to choose" but "which part of the work goes to whom."

The expert's role shifts from writing test cases to reviewing and correcting judgments. Reviewing a pass/fail with a one-line note takes seconds; writing a good test case from scratch takes minutes, and writing 200 is simply unrealistic.

Ye Yuan's team ultimately landed on a similar loop: evaluate at scale automatically, sample-review manually, conduct human review of every low-scoring conversation, and feed failure patterns back into the next iteration. Here's how it works in practice:

1. Automation explores the Agent and generates a first version of the test set
2. Experts sample-review, giving pass/fail plus commentary
3. Cases where experts and judges disagree are used to calibrate metrics
4. Calibrated metrics run automatically on every change
5. Experts only look at conflicts and newly emerging failure types

The human effort required is less than you'd think. Anthropic recommends starting with 20 to 50 tasks drawn from real failures. Hamel Husain starts with about 30 examples, continuously expanding until no new failure patterns appear, then accumulating roughly 100 labeled examples per failure pattern to validate the judge. Netflix continuously adds about 300 ratings per week because online traffic keeps shifting. This is a few hours of expert time, not a quarter's investment.

The Practical Loop: Automation Runs Tests, Experts Make Calls

In a Rhesis demo, I pointed the tool at a travel Agent with no requirements document whatsoever, let it explore and test on its own, then reviewed the results as a domain expert.

1. Architect explores the interface. In Quick or Comprehensive mode, it probes the domain from scratch, maps capability boundaries, and finds where the Agent refuses or escalates.
2. Architect plans and runs the test suite. Based on exploration results, it proposes requirements, metrics, and test set suggestions, waiting for approval before creating anything. I added a requirement myself: cancellation terms must match the fare rules. The cancellation test passed during the run.
3. Expert reviews results. Annotations can be attached to an entire test result, an individual metric, or a single turn in a conversation, each with pass/fail and comments. I marked the cancellation metric as failed and wrote the reason: the fare rules say 48 hours.
4. Human judgment takes priority. The expert's judgment now counts as the final result, automated judgments remain in the record, and Rhesis flags this disagreement as a status conflict, letting teams see where the judge and expert part ways. When the two agree, the result shows "Confirmed by" plus the reviewer's name, and over time you can see which metrics are trustworthy and which still need work.

All annotations collect in the Annotations hub under Improve, filterable by conflict, to fix metrics that keep failing. Engineers can also read and write annotations via SDK and MCP, keeping expert judgments and CI results in the same place.

Why status conflicts matter. In a hybrid evaluation loop, samples where expert and automated judgments disagree are the most informative. They either expose that the judge missed a business rule (like the 48-hour cancellation term) or show that the expert's criteria haven't been clearly articulated yet. Rhesis explicitly flags these disagreements as status conflicts and collects them in the Annotations hub filterable by conflict, essentially using "disagreement" as a calibration signal to actively hunt for metric flaws. Tracking how the conflict rate changes over time also tells you whether a metric is now trustworthy enough to hand off to automation for independent execution.

Let Experts Judge, Let Automation Scale

Neither approach alone is sufficient. Automated evaluation without expert input lets fluent but business-wrong answers slip through; human review without automation can only cover a few dozen conversations before stalling. Put experts where judgment is needed, and let automation handle exploration, attack, and repeated verification.

This logic has been cross-validated in production practices at Netflix, Anthropic, and Ye Yuan's team: the core of an evaluation system isn't the binary choice of "human or machine," but using human judgment as the anchor to calibrate automated metrics, then amplifying the calibration results to production scale through automation.

One-line Summary

AI Agent evaluation should let experts judge and automation scale, using human judgment to anchor automated metrics.

Paragraph Summary

The article argues that AI Agent evaluation can't rely solely on AI itself, nor solely on human labor. The core value of human experts lies in their grasp of organizational domain context—they can quickly spot business errors that generic metrics can't capture—but they're slow, expensive, and hard to scale. Automated evaluation excels at exploration without prerequisite information, large-scale adversarial testing, and continuous CI verification, yet LLM judges tend to be lenient, sensitive to wording, and high agreement rates mask missed detections at low failure rates. The pragmatic approach is to shift experts from writing test cases to reviewing pass/fail judgments, start with 20-50 real failure tasks, use disagreement cases to calibrate metrics, then let automation re-run them on every change. The Rhesis demo shows the complete loop of exploration, planning, expert review, and conflict flagging, and practices at Netflix, Anthropic, and Ye Yuan's team cross-validate this approach.