The Right Way to Skip Tests in AI Patches: Using Freeze Records to Catch Hidden Regressions

AI agent-generated code patches are increasingly common, but they bring a new twist on an old problem: when a test fails, is it a real bug, or just a flaky fluke? Model authors tend to simply quarantine the red test to make CI green — and in doing so, they may quietly hide a genuine regression.

The "freeze record" mechanism described in this article has one core idea: skipping a test must come with independently verifiable evidence, and this check must fail closed by default. If any field is missing, if both runs produce the same result, or if the decision-making oracle was modified by the same patch, the skip request is rejected.

Freeze Record Mechanism Overview

A note on scope: the original source explicitly states that the commands, checker scripts, and sample property files have not been executed. All hash values are placeholders, not real measurements. This article summarizes accordingly and should not be read as a run report.

What a Freeze Record Must Store

A valid freeze record has six mandatory fields — all of them required. An optional notes field may exist, but it can never substitute for any of the required fields.

Field	What It Stores	Condition for Rejecting the Skip
test_id	A single pytest node id	Zero or more than one test collected
fixture_path	Path to the snapshot fixture file	Path does not exist on disk
fixture_sha256	SHA-256 of the file's exact bytes	Digest changes between the two runs
disagreement	Two run objects with opposite passed flags	Both pass or both fail
oracle_path	Path to the property file	Path is missing at HEAD^
oracle_blob	Git blob id of the file at HEAD^	Blob differs, or the path appears in the patch

An expiration policy expressed in commit count can sit alongside these fields, but it does not itself authorize the skip. Reviewers still need to see both the disagreement and the parent-commit blob.

The record object should live at .freeze/record.json, so that CI and reviewers are always opening the same file. Do not bury these fields in a Pull Request comment — comments are not inputs to the checker.

SHA-256 is a cryptographic hash function that produces a fixed 256-bit digest from an input of any length. Its key property is collision resistance: it is computationally infeasible to find two different inputs that produce the same digest. The freeze record uses it to "nail down" the exact byte state of a fixture file — if even a single byte changes, the digest changes completely. A git blob id is essentially a SHA-1/SHA-256 hash that git uses to uniquely identify every version of every file stored in the repository. git rev-parse 'HEAD^:path/to/file' returns the blob id for that path in the parent commit, which corresponds one-to-one with the file's contents. Together, these two hashing mechanisms form the foundation of "tamper-evident" enforcement: any modification to either the fixture bytes or the oracle property file will cause a digest mismatch and trigger a rejection.

Why "Both Runs Agree" Is the Wrong Evidence

This is the most counterintuitive — and most critical — point in the entire mechanism. Many people would reason: if a test fails intermittently, just rerun it a few times, and once it's green, quarantine it. The original source explicitly argues against this.

Two passing runs only prove that the test can pass on this snapshot. They do not prove that the previous failure was non-deterministic. Two failing runs indicate a stable breakage — that's a product defect or a broken fixture, not a freeze candidate at all.

Only a "disagreement" constitutes the minimum valid observation: the result changed while the fixture bytes remained exactly the same. This is a deliberately weak signal — a bug that fails once in thirty runs will often produce two identical results consecutively, and will therefore never satisfy the condition.

This bias is intentional. The gate would rather let a rare flaky test stay red than quarantine a stable bug after one "lucky rerun." In other words, a freeze record should be treated as a brief, witnessed exception — not a probability estimate.

There is also a hard constraint: the snapshot must be a file that the test reads only. If the test rewrites the fixture, the second hash will differ and the record immediately becomes invalid. Tests that call the network, read a clock, or touch shared services are entirely out of scope — because a file hash cannot explain why they produce divergent results.

Flaky tests are tests that produce non-deterministic results — sometimes passing, sometimes failing — without any change to the code. Common sources include: dependence on the system clock or a random seed, shared mutable state between tests, asynchronous race conditions, network timeouts, and sensitivity to test execution order. Flaky tests are a persistent engineering governance challenge in CI systems: false positives consume engineer time, while simply quarantining them can mask real regressions. The freeze record mechanism's design philosophy is: rather than estimating how flaky a test is, require a minimal but verifiable piece of evidence — a confirmed result flip observed under a controlled fixture snapshot. This intentionally sets the bar high, making very low-frequency flaky tests (e.g., 0.1% failure rate) nearly impossible to satisfy the condition, thereby preventing stable bugs from being misclassified as flaky and hidden.

The Five-Step Numbered Process

The entire operation is broken into a strict five-step sequence. Stop at the first rejection, and do not add any skip marker until step 5 prints "accepted."

Before any test process starts, copy the fixture into .freeze/ and record its SHA-256.
Outside the test process, run the single test id twice and save each exit status.
Parse the blob of the property file at HEAD^ and confirm that path does not appear in the current commit's diff.
Run the copied fixture against the parent-commit property file; a property failure blocks the freeze.
Write record.json and run the checker — only an exit code of 0 allows the skip marker to be kept.

Fixing the Bytes and Capturing Two Runs

Step one "nails down" the bytes: first run sha256sum on the original file, then cp a copy, then hash the copy again before launching pytest to compare. If the two digests differ, stop immediately — you cannot proceed with a snapshot that is not reproducible.

Step two runs the same node id twice in separate processes — never with in-process retries. The original source specifically emphasizes that in-process retries will "hide" the very disagreement this record is meant to capture. Exit code 0 maps to passed; all others (after filtering out status 5) map to failed. Status 5 means no test was collected, which should be treated as a selection error — not a failed assertion, and certainly not a flaky signal.

Pinning the Oracle to the Parent Commit

Step three is the anti-cheating core of the entire mechanism: the property file must already exist in the parent commit, and the agent patch must not have edited it.

git rev-parse 'HEAD^:tests/properties/test_order_props.py' | tee .freeze/oracle.blob
git diff --name-only HEAD^ HEAD -- tests/properties/test_order_props.py

This split prevents a classic form of "model self-attestation": using the same change to both modify production code and rewrite the "witness" used to judge it. The diff list must be empty; if it is not, the freeze is rejected — even if the property run is green. A property file newly added by the patch is equally disqualified, because git rev-parse against HEAD^ will fail — let that failure surface honestly rather than catching it and continuing.

The original source also warns: during an interactive rebase, HEAD^ may point to the wrong parent commit, causing blob pinning to authenticate the wrong file. Always run the checker after HEAD has already been pointed at the patch commit.

Property-Based Testing is a testing paradigm distinct from traditional example-based testing, which specifies concrete inputs and asserts concrete outputs. Instead, it requires developers to describe invariants that a function should satisfy — relationships that hold universally across a class of inputs, such as "the length of a sorted list equals the length of the original" or "serializing then deserializing returns the original value." Testing frameworks (such as Python's Hypothesis) automatically generate large numbers of random inputs to attempt to falsify these invariants. In the freeze record mechanism, the oracle property file plays the role of an "independent witness": it describes a structural constraint unrelated to the specific patch, and therefore must exist before the patch. If the patch rewrites the oracle, the defendant is simultaneously serving as their own courtroom witness — which is exactly what blob pinning is designed to prevent.

Drafting the Parent Oracle: The Model Is Only an Assistant

Writing the invariants is the slowest step in the entire process. Someone must articulate a relationship that the fixture can support, then commit that file in a commit that an agent patch will not touch.

The original source discloses that the article is part of MonkeyCode product outreach, and that the free model access is positioned here solely as "a drafting aid for the parent file": request a candidate property based on a function signature and fixture schema, then review each assert before committing.

The key is the scope of use: the model draft should be attached in the review thread, and the oracle should be its own commit so that reviewers can diff the draft against the parent blob. Subsequent wording changes must also have their own commit — once they land in the agent patch, the blob pinning breaks and the checker rejects the freeze. And that rejection is the correct outcome. The model service is not treated as storage for fixture bytes; the bytes always stay in the git working tree, right next to the freeze record.

Known Limitations and Inapplicable Scenarios

This mechanism has clear boundaries, and the original source does not shy away from them.

Two samples cannot estimate a flakiness rate — they only detect that two results differed once. If you need probabilities, run a longer test campaign, and do not describe this record as such a campaign. The parent blob rule also does not evaluate assertion strength — a property committed earlier may be nearly tautological, so you must read the parent file yourself before relying on it. The recommended checker also does not run pytest --collect-only, meaning a malformed test_id may slip through; shallow clones with depth 1 have no HEAD^ blob to resolve — you must fetch the parent commit first, otherwise the job will fail closed, which is the intended default behavior.

Scenarios where this gate should explicitly not be used include:

When there is no parent oracle and no intention of committing one separately; snapshot suites that rewrite on every run have nothing for oracle_blob to pin.
Patches related to payments, authentication, or data loss — even if the disagreement is genuine, a freeze record is the wrong path; keep the test red and fix the production code.
When tests leak state between workers; shared contamination can flip results without changing the fixture hash, making the disagreement a "false witness."
When reviewers will not actually open record.json — an unread freeze file is just a skip comment with extra fields.

The final judgment still rests with humans. The checker enforces only the "shape" of the record; whether the invariants genuinely match the fixture is determined by the reviewer during the review. The reviewer needs to recompute the fixture digest and compare it against both disagreement entries, confirm that oracle_path is not in the diff, and open the parent property file to verify that its assertions describe a relationship that can be explained without reading the production diff. Only when both checks pass should that single test id be quarantined — and a follow-up task to delete the freeze should be scheduled immediately.

The value of this design lies not in any particular script, but in translating an engineering governance principle into an executable check: preventing AI patches from clearing themselves by rewriting their own evidence.