When an AI Coding Agent Says "Tests Pass," Which Files Did It Actually Test?

An Overlooked Problem: Which Version of the Code Do the Passing Tests Belong To?

When you run two coding agent windows on the same project simultaneously, one reports that tests pass, and the other completes a code review. Before you merge, someone makes a small edit — and the chat log still contains two reassuring messages, but neither one can tell you whether the files in front of you are the ones that were actually checked.

This is the core problem raised by the author of Squadra (an MIT-licensed open-source skill set and local Python toolkit for coordinating Claude Code, Codex, and interactive Cursor windows). Test passes and code reviews both belong to a specific version of the code. Once files change after inspection, the prior evidence no longer describes the current files.

This insight seems obvious, yet it is one of the most underrated aspects of multi-agent collaboration.

A Typical Runaway Scenario

Suppose the task is to fix an empty-state UI bug. Below is the illustrative workflow given by the author — not a benchmark or a real customer record:

Step	What Happened	What Was Established
1	Claude modified the empty-state handling logic	A candidate version exists to be checked
2	The agreed test command passed on that candidate version	Those checks passed on those files
3	Another Codex window reviewed the frozen candidate	A non-author reviewed that version
4	Someone edited a file that had been checked	The candidate version has changed
5	Tests and review are re-run on the new candidate	The task can be considered for closure based on current evidence

Step 4 is the most easily dismissed. That edit might be trivial — which is precisely the reason to check proportionally — but it does not automatically make the prior results describe the new files. A passing test is only valid for the code snapshot it ran against.

The practical consequence of this problem: the two messages "tests pass" and "review passed" in the chat window may still be there after the files have been modified, giving you a false sense of security. The merged code was never actually verified.

What Needs to Be Recorded Beyond the Chat

For handoffs between multiple agents, a useful record needs to answer several specific questions: who owns the task, what command was actually run, what results were returned, which files were checked, and who reviewed them.

Squadra's approach is to keep task assignments, acknowledgments, evidence, reviews, and decisions in a shared local registry. Agents follow a skill specification and use the CLI; a browser dashboard lets coordinators see the full task history.

The specific mechanisms include:

• The prove command: records check results and file fingerprints
• The deliver command: creates immutable review packages
• Pre-closure validation: before a task is closed, the candidate version, evidence, and review are re-checked together
• Invalidation mechanism: changes to source files invalidate prior evidence and reviews during the closure process

This last check is critical — even if the dashboard looks completely fine. The dashboard shows historical records, not a guarantee of the current file state.

File fingerprints are the core mechanism of Squadra's evidence chain and deserve a closer look. In software engineering, a fingerprint usually refers to a cryptographic hash (such as SHA-256) computed over file contents; any byte-level change produces a completely different digest. When Squadra's prove command records check results, it simultaneously binds the fingerprints of the files that were checked at that time. This way, even if a file is changed by a single character after the check, its fingerprint changes, and the closure process can immediately detect the mismatch during re-validation. Fingerprints do not record the historical contents of a file; they only identify the uniqueness of a specific version, making them both lightweight and reliable. Note that fingerprints only cover files explicitly included in the computation — untracked dependencies, generated artifacts, or external resources are not automatically included. This is the technical root of the article's emphasis that "source file fingerprints cannot prove everything about the runtime environment."

A Second Window Is Not the Same as an Independent Review

A key design constraint: the reviewer must have a different identity from the author. Giving the same window another alias does not make it independent.

Even when identity independence is satisfied, there remains room for judgment:

• Does the test cover the failure the task was supposed to fix?
• A passing test command may be insufficient
• An independent reviewer may also miss problems

Recording the handoff process makes it inspectable, but this does not prove the software is correct or ready for production. The author explicitly points out this boundary.

The same principle applies to scope. Ignored dependencies need their own evidence. A source file fingerprint cannot prove everything about the code's runtime environment — environment dependencies, system library versions, and configuration differences are all outside the coverage of source file fingerprints.

Identity independence is a stricter concept in collaborative systems than interface independence. Giving the same agent window two aliases or opening two tabs still means the same underlying model, the same context, and the same read/write permissions, so it cannot provide an adversarial perspective — it tends to confirm its own decisions rather than question them. True independent review requires the reviewer to be distinguished from the author in identity, session state, and even the context they rely on. Furthermore, independence is a necessary but not sufficient condition: even with different identities, both may share the same blind spots or misunderstandings about the task objective. This is what the article means by "recording the handoff process makes it inspectable, but this does not prove the software is correct" — inspectability means the evidence can be traced and reviewed, but it does not mean the conclusion is necessarily valid. Mistaking inspectability for correctness is a common cognitive trap in automated workflows.

How to Get Started

It is recommended to start with a small change that has clear check criteria, then track one task through the complete process of assignment, evidence, review, and closure. This lets you judge whether the extra recording overhead is worthwhile before applying it to a larger project.

Installation and runtime requirements:

• Python 3.12+ and Git (required)
• Node.js: needed for the dashboard checks in the self-tests
• Test environment: macOS/POSIX
• Cursor: works in interactive mode
• There is no Squadra account system, and your coding agents keep their usual costs

The repository includes installation instructions, screenshots, and an 80-second animated demo.

The Core Tension in Multi-Agent Collaboration

What Squadra solves is essentially an evidence freshness problem. In a single-agent, single-window workflow, there is usually no other modification between tests passing and code merge, so the chat log is sufficient evidence. But when multiple agents work in parallel, different windows read from and write to the same project, and the probability that file state changes between check and merge rises significantly.

The solution is not to make agents smarter, but to explicitly bind evidence to code versions. File fingerprints, immutable review packages, and pre-closure re-validation — the common goal of these mechanisms is to ensure that the conclusion "passed" always points to a specific, traceable set of files.

At the end, the author raises a question worth thinking about: if you use multiple coding agents, what do you need to see in a handoff before you trust a review — a diff, command output, an independent reproduction, or something else?

Evidence freshness draws on the idea of optimistic concurrency control in distributed systems. In a multi-replica environment, the state seen at read time may already be stale before write time, so systems need version numbers, optimistic locks, or checksums to detect conflicts. Squadra's pre-closure re-validation is essentially moving this idea into the agent collaboration workflow: instead of assuming evidence remains valid indefinitely, it re-verifies at the moment the evidence is used whether it still points to the current version. Unlike databases, the "write" here may be a casual human edit or a parallel change by another agent, making the triggers more numerous and more hidden. Understanding this helps you generalize — any "check first, merge later" process, as long as concurrent writes are possible in between, needs similar version-binding protection; otherwise the check conclusion will quietly lose its meaning.

One-line Summary

In multi-agent collaboration, test passes and code reviews are only valid for a specific version of the code; once files are modified, prior evidence becomes immediately invalid.

Paragraph Summary

The article focuses on an underrated problem in multi-agent coding collaboration: test passes and code reviews both belong to a specific version of the code. Once files are modified after inspection, the "passed" messages in the chat log can no longer describe the current files, creating a false sense of security. Squadra explicitly binds evidence to code versions through a shared local registry, the prove command for file fingerprints, deliver for immutable review packages, pre-closure re-validation, and a change invalidation mechanism. The article also emphasizes that independent review requires the reviewer's identity to differ from the author's; aliasing the same window does not count as independent. It also points out that source file fingerprints cannot cover runtime conditions such as environment dependencies. The final conclusion is: the solution is not to make agents smarter, but to make evidence traceably point to specific files.