The AI Coding Era: Replacing Manual Code Review with Evaluation Gates

The Core Tension: Code Gets Written Faster Than Anyone Can Read It

Now that coding agents like Claude Code and Codex handle a large share of code writing, a long-overlooked problem has come into sharp focus: how do you ensure quality when code is being generated faster than anyone can read it?

Based on data from the Zoroval team, the median agent-generated PR touches only 133 lines — not enormous on its face — but roughly one in every eight changes modifies more than 1,000 lines of code. What makes this even trickier is that an engineer in their first week on the job can submit a PR that edits a prompt or context constructor. Writing that change has become trivially easy, but judging its actual impact on the product is just as hard as it's always been.

Faced with this tension, two obvious answers both turn out to fail. Answer one: review every line of code. This has been the gold standard of code review for decades, but no one can seriously review thousands of lines of agent-generated code multiple times a day — so people start skimming. Skim reviews look like reviews but catch far fewer problems than you'd think, and they turn the reviewer into the slowest link in the entire system. Answer two: block PRs until someone has bandwidth. This is safe, but it slowly kills the team — if every change waits days for review, you're not actually using agent coding at all; you've surrendered the speed advantage entirely.

Zoroval chose a third path: for changes that alter LLM behavior, build an "evaluation gate" — because those changes are precisely the ones that reading code cannot verify.

Code review can't keep up with agents. Gates can.

A Silent Failure: How a Context Constructor Quietly Broke the Judge Model

The whole system grew out of a real incident. Zoroval's product uses LLM-as-a-judge: one model reads conversation traces and decides whether a named "failure pattern" is present. Customers' monitoring depended entirely on this judge.

On July 13th, the judge missed every failure that had been confirmed by humans. The monitoring job ran 1,138 judge calls. The golden dataset contained four human-confirmed failures. The judge answered "no" to all four — and the entire run reported zero errors.

The model hadn't changed. The prompt template hadn't changed. The team's evaluations only triggered when one of those two things changed, so no one had run the golden dataset. The actual root cause was the context constructor — the code responsible for assembling what the model actually sees. It's just as much a part of an intelligent system as the model and the prompt, but changing it never triggered an evaluation.

This exposed a core problem with LLM systems in CI: code tests check the shape of outputs (e.g., is this valid JSON with the right fields?) but cannot check judgment. A code change surrounding a model can pass every test while completely changing the model's conclusions. With coding agents in the mix, these kinds of changes happen constantly — often in a PR that was never supposed to touch the model at all. What the team needed was an evaluation gate that knew which files change model behavior and refused to let them through on a green test suite alone.

LLM-as-a-judge is an approach that uses a language model to automatically evaluate the output quality of another model or system. Because human annotation is expensive and slow, LLM-as-a-judge is widely used in monitoring, testing, and evaluation pipelines: the judge model receives conversation logs or model outputs and assesses quality, detects failure patterns, or assigns scores according to predefined criteria. Its core vulnerability is that the judge is itself a model — influenced by the prompt, context, model version, and other factors — and those influences rarely surface as errors. They just quietly change the verdicts, as the incident above illustrates.

Context constructor refers to the code layer responsible for assembling, formatting, and filtering inputs before they are fed to a model. It determines what the model ultimately "sees": which conversation turns get truncated, how retrieved documents are ordered, what format metadata is injected in. Changes to a context constructor don't touch model weights or prompt templates, yet they can fundamentally alter the model's reasoning premises. The context constructor is one of the most easily overlooked sources of behavioral change in LLM systems — and one of the least likely to be caught by conventional tests.

The Core Principle: Review Evidence, Not Lines of Code

Agents make changes cheap. We require that those changes have current evidence before reaching production.

This sentence is the heart of the entire LLMOps approach. The team doesn't try to stop LLMs from making mistakes — agents writing code and models inside the product will both make mistakes — but they build an environment where mistakes are cheap, visible, and caught before they reach users. Four principles run throughout:

Review evidence, not lines of code. Humans still make decisions, but those decisions are based on test results, evaluation scores, fingerprints, and precise commit SHAs — not a 1,000-line diff. Security, architecture, and interactions between changes still require human judgment.
Humans own the rules. The policy files that define gates are small. Modifying them requires review from an owner (a senior or ML engineer), and the gates apply those rules to every change.
Gates talk to agents. When a gate fails, its output reads like a ticket: it explains what changed, why it matters, and how to fix it — so a coding agent can resolve it on its own. Most CI output is written for humans; this output is written for the next agent iteration.
Approvals are bound to exact content. Every approval points to a fingerprint or SHA. Change one more line and the approval automatically becomes invalid — no one has to remember to re-check.

How Evaluation Gates Work in CI

The check known as the "Intelligence Change Gate" is the central idea of this entire framework. It operates in five steps.

Step 1: Declare the Protected Scope

A policy file lists all paths that could change model behavior — prompts, context constructors, model settings, retrieval code, and so on — grouped by the LLM component they affect (called a stage in the policy, e.g., monitoring). Worth noting is the line_patterns field: settings files change frequently, so the gate only triggers when the specific line that configures the monitoring model or inference intensity changes — it won't fire false positives when someone adds a database timeout.

Step 2: Fingerprint the Exact Change

For each PR, the gate identifies which protected files were modified, takes only the diff of those files, and SHA-256 hashes that diff together with the affected stages and rules to produce a change fingerprint. If someone else modifies the same file and you rebase, the fingerprint changes and new evidence is required. This is intentional: the code you measured is no longer the code about to be merged.

SHA-256 is a cryptographic hash function that produces a fixed-length (256-bit) digest string for any input. Its key properties are determinism (the same input always produces the same hash) and the avalanche effect (even a single byte change in the input produces a completely different hash). Here, SHA-256-hashing the diff of protected files means that any change to a single protected line of code changes the fingerprint, automatically breaking the match with any prior evaluation report. This makes "evidence bound to exact code" a machine-enforceable property — not something that depends on engineers' memory or process discipline.

Step 3: Require an Evaluation Report Matching That Fingerprint

If protected files have changed, the PR must include an evaluation report: a small JSON manifest listing the fingerprint, the golden dataset, the baseline, the candidate, and per-stage results. The gate doesn't trust the report's passed field — it recalculates pass/fail from the raw numbers. The underlying data stays private; the report references it only via hash. No customer data ever enters the repository.

Step 4: Let Evidence Expire Automatically

This is the team's proudest design decision. Because the fingerprint covers the exact protected diff, any edit to a protected file after the evaluation runs — whether from a human or an agent — changes the fingerprint. The old report no longer matches, and the gate fails again. Changes to documentation and tests don't affect the fingerprint, so fixing a typo still works. No one has to remember that "this evidence was for the old version" — the gate remembers for you.

Step 5: Preserve Visible Broken-Window Overrides

Sometimes you consciously accept a tradeoff. For instance, if a failure pattern's target shifts from precision to recall, the evaluation will show precision dropping. Rather than tweaking thresholds to make it look like a pass, you record it as a decision. The gate allows owner overrides, but each override is a standalone file scoped to a single PR number. It must specify the exact fingerprint, include a rationale and risk statement, come from an approved reviewer on the owners list, expire after seven days, and always appear in CI as owner_override — never as a plain pass. On the team's staging branch there are 43 evaluation reports, of which 9 PRs used an owner override.

What does the agent see? When a gate fails, the output it prints is essentially a ticket: the agent knows which stage to evaluate, which files caused the failure, and which fingerprint to write into the evaluation report. Sometimes the agent realizes it shouldn't have touched that file at all, reverts that part of the change, and the gate passes without needing an evaluation — which means the gate also catches accidental scope creep and misunderstood requirements.

What Passing the Evaluation Gate Doesn't Prove

Gates don't just block bad changes — they also make good changes feel safe to try.

The team tested a judge model that was roughly six times cheaper. It found all known failures in the golden dataset on the first pass and sailed through. But in a broader comparison, it said "yes" to a particular failure pattern far more often than the existing judge — because it had dropped part of the definition and was flagging cases that were out of scope.

The first round of checks missed this because the golden dataset contained only positive examples. That kind of dataset measures recall (can the model find failures?) but never measures precision (when it says "yes," is it right?). A model that over-answers "yes" simply cannot fail on such a dataset.

The cheaper model later went to production after passing an evaluation that included both positive and negative examples, reducing monitoring costs by roughly 83%. Its evaluation report came with a caveat: one failure pattern had no positive examples, so recall remained untested. The gate doesn't design the sample — it only ensures that evidence for the exact change exists and that its limitations are written alongside it. Humans decide what the evidence must cover, because LLM changes fail in ways that no single number can capture.

This also draws a clear line between what the gate can and cannot prove. It can verify that protected files actually changed, that the report matches the exact diff, that each stage has a quality result, that the sample meets the minimum size specified in the policy, that the run is recent enough, and that each result meets the thresholds stated in the report. It cannot verify that the evaluation actually ran, that the numbers are real, that the thresholds are appropriate, or that the golden dataset covers the failure patterns that actually matter. That is why every evaluation report still requires human review.

The Four-Layer LLMOps Pipeline Around the Gate

The evaluation gate works because of the full system surrounding it. The team's gates span four layers — local, PR, staging, and production — and a change must pass all of them.

The local layer is the cheapest. Agents can catch errors before a PR is even opened. The repository root contains an AGENTS.md that Claude Code, Codex, and similar agents load before starting work. It describes boundaries, not coding style — for example: "develop is staging, main is production" and "never push to main, never bypass a failing guard." These rules are kept short and absolute, because experience shows that agents follow clear rules far more reliably than they follow nuanced paragraphs. Each agent also works in its own independent git worktree, preventing parallel agents from overwriting each other.

The PR layer runs Product CI. In addition to standard backend tests, frontend checks, and database contract tests, two components are specific to LLM systems: the evaluation gate described above, and regression evaluations for the code surrounding models. Regression evaluations run on every PR using frozen synthetic cases and recorded LLM inputs and outputs — no real model calls are made, so they're fast, free, and verify that behavior around the model remains consistent. It's also important to protect the gate itself: an agent that can edit the gate can turn it off. So CODEOWNERS entries should cover the policy files, scripts, workflows, evaluation reports, and override folder, with "Require code owner review" and "Required status checks" enabled on GitHub. A small-team gotcha: GitHub doesn't allow PR authors to approve their own PRs — if an agent opens PRs from your account, it needs its own separate account.

The staging and production layers have one iron rule: merging to staging never authorizes a production release. The team never merges develop into main. Production is the only place where a human decision is required every single time. Releases use a selective approach: the owner specifies which PRs to release, the release branch starts from the current main and carries only those PRs' squash commits. Agents preparing a release post a one-page "evidence package" containing PR numbers, exact head SHAs, migrations, evaluation reports, config changes, LLM cost impact, and test results. Approval is a PR comment bound to the exact commit — any new commit invalidates it. Here, LLM cost is part of correctness: a change that doubles the bill is a regression, even if every individual answer is right.

Git worktree is a Git feature that lets the same repository check out different branches into multiple directories on the filesystem simultaneously. Each directory has its own working area and staging index, but all share the same .git object store. In scenarios where multiple agents work in parallel, if they all share a single working directory, their file edits and staging operations will interfere with and overwrite each other. By giving each agent its own worktree, they operate in logically isolated sandboxes in parallel — without the conflicts, and without needing to fully clone the repository for every task, saving disk space and initialization time.

The team honestly lists the limitations: it adds friction (changing one line of a context constructor might still require running a live evaluation); the quality of the evidence depends on the golden dataset (50 cases may miss rare failures); evidence is self-reported (CI never touches private data — it only checks hash formats); the policy may be incomplete (files not listed are completely invisible to the gate); it doesn't replace security or architecture reviews, nor does it replace production monitoring. Most importantly, it has only been validated on a small team and has not yet been run at the scale of a fifty-person organization.

If you want to get started, you don't have to implement the whole system at once: first build a small golden dataset from real traces that includes both positive and negative examples; list the files that can change model behavior (be sure to include context constructors); add a required CI check that fails whenever those files change; add a line about LLM cost impact to your PR template; and rewrite one CI failure message as a ticket an agent can act on. Zoroval has packaged these gates as an open-source agent skill.

One-line Summary

Zoroval uses SHA-256 fingerprints to bind CI evaluation reports to exact diffs, forcing AI coding agents to submit behavioral evidence before merging LLM-related changes.