Human-in-the-Loop QA Workflow: Adding Test Gates to Coding Agents

Why Coding Agents Need a Human-in-the-Loop QA Process

Getting AI agents to write code is no longer novel, but ensuring that the code they write actually meets requirements is still a thorny problem. A developer on Reddit shared a human-in-the-loop QA workflow that attempts to constrain coding agents through a "write tests first, human approval, then script-based adjudication" approach. The core idea isn't complicated: have the agent write tests before the feature exists, have a human approve them once, and finally use a script—not the agent—to determine whether the tests pass.

This article breaks down the specific design of this workflow, its measurement methodology, and the lessons the author learned from real testing, and discusses its implications for AI programming practice.

Workflow Breakdown: Three Roles and One Human Gate

The Test-Writing Agent: Fail First, Then Freeze

The author uses Claude Sonnet (via Claude Code) as the "writer" role. For each acceptance criterion, it needs to write a unit test (Vitest) and a browser test (Playwright). The key constraint is: each test must first fail on its assertion, so that it cannot write invalid tests that "pass no matter what the code does."

After the tests are written, the process enters the human stage: a person approves these tests on the Pull Request, and each acceptance criterion must come with a justification for approval. Once approved, the tests are frozen—a hash is computed, and branch rules reject any subsequent modifications. This effectively puts a lock on the tests themselves, preventing the agent from going back and tampering with the tests during the implementation phase to "force a pass."

Vitest is a JavaScript/TypeScript unit testing framework built on Vite, characterized by fast startup and seamless integration with Vite's module resolution and transformation pipeline, making it suitable for quickly verifying logic at the Node environment or component level. Playwright is Microsoft's open-source end-to-end (E2E) browser testing tool that can drive Chromium, Firefox, and WebKit to render real pages, supporting auto-waiting, multiple tabs, network interception, and more. The typical division of labor between the two is: Vitest covers pure functions, state logic, and component contracts, while Playwright covers interactions and accessibility behavior in real browsers. The dual-layer design of "unit tests + browser tests" in the article is precisely about having the agent face acceptance criteria at both the logic layer and the UI layer, avoiding blind spots that arise from writing only one type of test.

Test freezing refers to computing and recording a hash of the test content after it has been manually approved, then rejecting any subsequent changes through CI branch protection rules or hooks. Its purpose is to cut off the agent's shortcut of "changing tests to force a pass": if tests can be modified at will, the agent could loosen assertions or delete failing cases to make the implementation appear correct. The hash value acts as a fingerprint of the test version—once the hash of a submitted test file doesn't match the approval record, the pipeline fails. This mechanism puts "tests as contracts" into practice, turning tests from editable drafts into protected acceptance baselines.

The Verifier Agent: Read-Only, No Writing

The verification stage is handled by Claude Haiku, which has no write permissions and is only responsible for checking the completed code against the frozen tests. But the final say does not rest with it: a script compares its judgment with the actual test report and re-runs the entire test suite. In other words, the agent provides "opinions," and the script provides "facts."

Tool use is also strictly fenced: allowlisted commands, guard hooks on every tool call, a network-free sandbox environment, and per-run budget caps ($1.00 for the writer, $0.20 for the verifier).

Read-only permissions in agent design reflect the principle of least privilege: the verifier can only read code, tests, and reports, and cannot write files, change configurations, or execute commands that alter repository state. This way, even if the verifier makes incorrect judgments, it cannot pollute the codebase or tamper with tests. Combined with a sandbox environment (no network, restricted filesystem) and guard hooks (allowlist checks on every tool call), the agent's behavior can be confined to an auditable scope. Budget caps provide a second safeguard from the cost dimension, preventing runaway loops from consuming resources.

The Human Gate: The Most Error-Prone Link

The author specifically points out that the human stage is "the easiest part to get wrong." On two real tickets, both sets of tests were merged before anyone approved them. This detail shows that no matter how well the process is designed, if human approval is not enforced, the entire gate becomes nominal.

Measurement Methodology: 17 Tickets and Hidden Tests

To validate the effectiveness of this workflow, the author set up 17 small tickets on a demo store project. Each ticket had hidden tests, a reference implementation, and pre-planted bugs. The agent ran on an isolated CI runner and could not see these hidden tests at all. Each ticket was run 3 to 6 trials, and results were tallied from the test reports.

The data shows: the current version passed 21/21 trials on the 6 tickets it ran, capturing 33/33 pre-planted bugs; an earlier version passed 49 out of 51 trials across all 17 tickets. The author also acknowledges that the sample size is small and that these are "signals, not proof."

Four Lessons from Real Testing

"Writing More Test Cases" Backfires

Intuitively, writing more tests should catch more bugs. But the actual results were the opposite: increasing the number of tests did catch a bug that was missed 8 times out of 15 trials, but at the cost of more correct code being incorrectly judged as failing—2 occurrences out of 48 trials, rising to 5 afterward. The more tests, the more false positives.

Having the Agent List "Uncovered Scope" Is More Effective

Rather than piling on more tests, asking the agent to explicitly list what its tests do not cover works better: that bug was caught 5 out of 5 trials. This makes the agent proactively expose its own blind spots rather than blindly expanding test coverage.

Test coverage usually refers to the proportion of code lines, branches, or functions executed by tests, but it only indicates "executed," not "asserted." Agents can easily write tests with high coverage but low assertion strength, so chasing coverage numbers alone is misleading. Having the agent explicitly list "uncovered scope" is essentially asking it to perform a coverage gap analysis: identifying which acceptance criteria, boundary conditions, and error paths have not yet been touched by tests. This self-reported list is closer to the semantic level than a coverage report and makes it easier for humans to judge during approval whether the tests are sufficient, turning blind spots from implicit to explicit.

The Agent "Guesses" Nonexistent Structures

The writer agent makes incorrect assumptions about UI structures that don't yet exist. For example, it guessed the "textbox" role for <input type="search"> (actually "searchbox"), or wrote text with a total as an isolated "€9.40" instead of "Total €9.40." The author fixed this with two specific rules: find fields by label, and check the value as part of the element's text.

Rejected Tool Calls Consume Turns

When a tool call is rejected by a guard, it consumes a turn each time. The author found that if the rejection message explains what should be done instead, the writer's rejection count between two activities dropped from 16 to 1, and the average number of turns dropped from 11.3 to 9.0. Rejection is not just interception—it should be guidance.

Implications for AI Programming Practice

The value of this workflow doesn't lie in how powerful the models it uses are, but in how it assigns the deterministic parts to scripts, the judgment parts to humans, and the generative parts to agents. The test-first constraint, the test-freezing mechanism, and the verifier's read-only permissions are all engineering means of limiting the agent's degrees of freedom.

The question the author poses at the end is worth considering for anyone using coding agents: Where would you put the human in the agent workflow? Based on this real-world test, the answer is likely—after the tests are written, before the code is written, and make sure that position cannot be bypassed.

One-line Summary

A human-in-the-loop QA workflow where the agent writes failing tests first, humans freeze them, and scripts adjudicate—plus lessons from real testing.

Paragraph Summary

The article breaks down a human-in-the-loop QA workflow proposed by a Reddit developer: the writer agent first writes failing unit tests and browser tests for each acceptance criterion; after human approval on the PR and hash-based freezing, a read-only verifier agent checks the code, and finally a script re-runs the test suite to make the factual ruling. The author measured effectiveness using 17 tickets with hidden tests and pre-planted bugs, finding that writing more test cases actually increases false positives, while having the agent list uncovered scope is more effective; agents guess at nonexistent UI structures and need specific rules to constrain them; and attaching guidance to rejected tool calls can significantly reduce turns. The conclusion is to give determinism to scripts, judgment to humans, and generation to agents, while ensuring the human gate cannot be bypassed.