The Hidden Risk of AI Code Review: Why Model Failover Can Approve Unreviewed Patches

When an AI agent handles code review inside a CI pipeline, a seemingly harmless engineering detail can cause the system to approve a patch that was never actually reviewed. This lab postmortem documents a specific failure scenario: after a model failover, a stale evaluation receipt was reused, causing the gate to pass a change that the new model never scored.

The Core Problem: Receipts Decoupled from Reviewer Identity

The key takeaway can be stated in a single sentence: an evaluation receipt must be bound to the diff, prompt, model, and workspace generation it corresponds to. A matching diff hash alone does not constitute a valid review.

The failure chain is straightforward. The experiment ran two lab models (lab-model-a and lab-model-b) in a recycled workspace. The flow went roughly like this: a stale receipt was copied into the recycled task directory; model A wrote a passing receipt for the diff; then model A went offline, and the retry logic selected model B; model B found a receipt.json already on disk and skipped scoring entirely.

The critical point: failover treated "file exists" as "review complete." The receipt on disk was a product of the previous attempt. The new model never re-evaluated. This isn't a cache invalidation issue or a lock-file retry problem — the receipt was perfectly legitimate for the other attempt; it simply didn't belong to this merge.

Workspace generation is one of the central concepts in this fix and deserves its own explanation. In CI systems, the same server or directory path can be reused by multiple tasks in sequence — after one task fails, the system may or may not clean up, then reassign the same directory to the next task. A "workspace generation" is a monotonically increasing or randomly generated identifier introduced to distinguish these successive lifetimes: every time a workspace is reassigned to a new task, a new generation value is minted, and any leftover files carrying the old generation are rejected at verification time. This mechanism is conceptually identical to optimistic lock versioning in databases — it doesn't prevent stale data from being written to disk, but it ensures stale data is identified and rejected at read time. Without workspace generations, two tasks sharing the same diff hash are indistinguishable at the filesystem level, creating the conditions for receipt reuse.

Why a Naïve Gate Gets Fooled

The naïve check shown in the postmortem is extremely simple — and dangerous precisely because of that simplicity:

def naive_accept(receipt, diff_sha256):
    return receipt.get('diff_sha256') == diff_sha256 and receipt.get('verdict') == 'pass'

This function compares a single stored hash and stops there. The diff bytes match, so it returns success. It completely ignores the model ID, the workspace generation, and whether any review reasons were stored. On a recycled server, single-field checks like this are trivially deceived.

The postmortem lists several compounding factors that engineering teams should audit against: receipts used a fixed filename shared across all attempts; workspace reuse preserved scoring files from before the failover; the gate trusted the diff hash but didn't verify scorer identity; the agent and gate shared the same writable server directory; there was no attempt ID linking task logs to the approved receipt; and cleanup logic only ran on success, so crashes left stale files behind.

The Persistent Fix: Bind Identity, Don't Just Extend Timeouts

The fix isn't about making the model "try harder" — it's about refusing to vouch for a receipt from a different attempt. The postmortem provides a program called bind_receipt.py (targeting Python 3.9+ in a standard CI image) that strictly binds the receipt to four fields: diff, prompt, model, and workspace generation.

The core mechanism: the bootstrap mints the workspace generation before any agent starts; the expected model ID is captured when the model is selected; the gate recomputes the hash and compares every bound field. Any missing field is treated as task failure, not a soft warning.

The program exposes three subcommands: write, verify, and selftest. The postmortem explicitly emphasizes that the self-test must be run before wiring this into CI:

python3 bind_receipt.py selftest

A failed self-test means the binder itself is untrustworthy. The verify command prints any mismatched field names and returns exit code 1; a field mismatch fails the task immediately with no retry.

Decision Table: Naïve Gate vs. Bound Gate

The postmortem uses a decision table to clearly contrast how the two gates behave across various states:

Observed state	Naïve gate	Bound gate
Same diff, new model ID	Accept	Reject
Same diff, new workspace generation	Accept	Reject
Same identity, verdict is fail	Reject	Reject
Same identity, reasons is empty	Accept	Reject
Diff bytes changed after scoring	Reject	Reject
Receipt missing	Depends on script	Reject
Fresh matching receipt with reasons	Accept	Accept

Only the last row is a valid merge path; all other observed states keep the gate closed and block the merge. The naïve gate incorrectly accepts in exactly the cases that should be blocked — new model ID, new workspace generation, and empty reasons — which is precisely the root cause of this incident.

Bootstrap Rules and the Free-Server Work Contract

The postmortem repeatedly emphasizes one principle: these rules belong in the job image, not in the prompt. An agent under review has the ability to edit prompt text.

Specific bootstrap rules include: mint WORKSPACE_GEN in bootstrap before the agent process starts; export MODEL_ID once the model is selected and freeze that value; store each receipt in a directory named by attempt ID; ignore any receipt paths left over from a previous generation; fail the task without retry on any non-zero return from verify; compare the model ID in the task log against the receipt before merging; treat a missing receipt as failure, not as a skipped check.

On a "free server" (a reusable environment with switchable scorers), bootstrap should also wipe the agent-writable scoring path — even when the binder is already in place, because dirty directories can hide other files the agent might mistakenly trust. The four bound values should be passed as command-line arguments, not as prompt text.

Why rules must live in the job image rather than the prompt maps directly to a classic principle in security engineering: the policy enforcement point must sit outside the reach of the subject being constrained. A prompt is text the agent can read, reference, and in some tool-call scenarios indirectly influence at runtime. A job image is an environment that the CI platform locks in before the agent process starts — the agent cannot modify binaries already compiled or installed in the image. This is the same logic as "user-space processes must not be able to modify the kernel." Placing gate logic in the image effectively moves the trust anchor outside the agent's control surface, making "agent rewrites review rules to pass itself" structurally impossible rather than merely discouraged by convention.

Limitations You Must Acknowledge

The postmortem candidly lists its method's boundaries — something engineering teams should value even more than the solution itself.

Hashes do not measure review quality. A matching receipt may still describe a cursory read of the diff — the gate only checks identity and the presence of reasons. A hostile runner can forge every bound field in the file, so the binder is not an attestation service against model vendors; it must be used alongside runners the team already trusts.

The workspace generation must come from bootstrap, not from the agent. If the agent mints this value itself, it can rewrite attempt history, and the gate will vouch for whatever story the agent fabricates. Also, the self-test never calls a remote model — a green self-test does not mean the patch has been reviewed.

The postmortem also explicitly states scenarios where this approach is not appropriate: when the team needs vendor-signed proof of model execution (this script cannot provide that); when there is no independent release policy (the binder is a CI check, not a release approval); when workspaces are always destroyed after each task and model IDs never change between attempts (a simpler diff hash may suffice); and for quota planning, cost forecasting, or hardware selection (consult vendor documentation for those operational numbers).

Engineering Takeaways

As AI agents become increasingly embedded in CI pipelines and take on review responsibilities, implicit assumptions like "file exists means task complete" become real security vulnerabilities. The value of this postmortem lies not in the Python code itself, but in identifying a widely overlooked failure class: the decoupling of review artifacts from reviewer identity.

It's worth noting that the original article discloses it is part of MonkeyCode product outreach content, and the "free model access" and "free servers" mentioned are assumptions provided by the operator, with the author explicitly stating no availability details were verified. But even setting aside that commercial context, the core engineering lesson holds: any host that recycles workspaces is exposed to the same vulnerability. Placing the binder in the job image and blocking merges on field mismatches is a low-cost, high-value defensive measure.

The methodology in one sentence: bind receipts before failover can replay old scores, mint workspace generations outside the agent process, and fail closed whenever the model, prompt, or diff don't match.