Treating Agents as Colleagues: Engineering Practices for Validation, Architecture, and Auto-Merging PRs

When engineers start letting agents auto-merge PRs, and even wake up to find 20 PRs already in the main branch, it's not blind trust—it's a carefully designed system of validation and constraints. In Growth Bar's practice, an engineer who submitted over a thousand PRs in five months shares the path from "micromanagement" to "engineering manager."

The Trust Curve: From Watching Every Output to Letting Go in Parallel

When first using agents to write code, most engineers fall into a tight loop: enter a prompt, watch every output, check the code, find problems, and give feedback again. This mode makes it impossible to parallelize tasks, because you are always the validation bottleneck.

The speaker used a hand-drawn diagram to describe this shift in mindset: at first he only dared to bind one or a few agents, personally confirming everything; as trust in the agents grew, he gradually dared to launch more tasks simultaneously. Over the past five months, his trust curve has risen steadily, and now he allows agents to auto-merge pull requests. When he woke up that morning, 20 PRs had already been merged into the main branch, and the quality was genuinely good.

The core analogy for this shift is engineering management: if you don't trust your team members, you become a micromanager, standing behind each person and repeatedly checking. Only when you build trust can you act like an engineering manager—designing the environment and assigning tasks, rather than personally writing every line of code.

Validation Skills: Let the Agent Actually Run Its Own Code

The most critical step in building trust is "validation." An agent cannot just write code and declare itself done; it must be able to run the application—collect performance traces, take memory snapshots, open the simulator, and personally confirm whether the functionality works. This doesn't guarantee elegant code, but at least it guarantees functional correctness.

Take the agent window internally codenamed Glass as an example. In his first week on the job, the speaker faced release pressure and could only open Chrome DevTools himself to collect flame graphs, while the agent was equally clueless—confidently pointing to the wrong location, and often inaccurately. So he packaged the ability to control the Chrome DevTools Protocol, launch the application, and collect performance data into a skill, letting the agent operate autonomously.

But operational capability alone isn't enough. Once the app launches, the agent doesn't know where to click or how to reach a specific feature, and can only search around randomly. To solve this, he introduced a "feature map"—a file that clearly records each feature's entry point, keyboard shortcuts, and DOM element attributes. When a user report is so vague that it's just a screenshot plus three question marks, the agent can use the map to understand the context rather than answering "I have no idea."

This capability was later packaged into the pstack plugin (a joking homage to Gary Tan's gstack), which includes a tool for creating validation skills that can automatically explore the code to generate a feature map and continuously maintain it as the application changes.

Chrome DevTools Protocol (CDP) is a WebSocket-based remote debugging interface that allows external programs to programmatically control the Chrome browser: opening pages, executing JavaScript, capturing network requests, collecting performance traces and heap snapshots, and more. For an agent, CDP is essentially a pair of "hands" that can operate the browser—it doesn't need to rely on a human manually opening DevTools to collect flame graphs, but can directly invoke protocol commands to launch the app, record performance data, and locate lagging functions. Once CDP is packaged as a skill, the agent gains closed-loop validation capability: write code, run the app, collect data, and determine whether functionality works, thereby escaping the blind spot of "writing without validating."

Evaluation Manual: Unit Testing Your Skills

Once written, a skill isn't set in stone. The speaker emphasizes that maintaining skills requires continuous judgment and taste, like pair programming—sitting next to a colleague and constantly suggesting improvements.

The method he uses is called "evaluation," which is essentially unit testing for skills. The specific approach: launch multiple sub-agents; the main coordinating agent first defines the scoring criteria, and the sub-agents execute tasks in independent directories—the directory naming is deliberately designed to avoid letting sub-agents realize they are being evaluated, because agents change their behavior once they discover they are being tested. pstack supports multiple models, so the same skill can be run on different models for comparison, with particular attention to the model combinations actually used day to day.

Evaluation is highly iterative: the coordinating agent can score, then judge agents from different models can be brought in to cross-check, avoiding single-model bias; a loop command can also be used to run repeatedly until all items receive full marks.

Sub-agent refers to an agent instance spawned by a main coordinating agent that executes a specific task in an independent context. Each has its own conversation history and working directory, isolated from the others, so multiple test cases can be executed in parallel without contaminating each other's context. In evaluation scenarios, the main coordinating agent defines the scoring criteria, assigns tasks, and aggregates results, while sub-agents complete tasks in isolated environments like "exam candidates" being tested. This architecture both improves the parallel efficiency of evaluation and avoids the problem of a single agent forgetting early instructions in a long conversation—a common task decomposition pattern in multi-agent systems.

Architectural Constraints: Make the Fastest Path the Best Path

When agents can auto-merge PRs, what safeguards codebase quality? The speaker's answer is a layered constraint system.

The first layer is the structure of the codebase itself: concepts such as feature entry points and conversation record cards have clear names and fixed creation methods, and a complete feature lives entirely in one directory. Agents love to copy existing patterns, so clear conventions are the strongest constraint. The second layer is static analysis and continuous integration checks, turning rules into hard failures—for example, in Growth Bar's architecture, useEffect is directly prohibited, and even writing code comments is prohibited, because 99% of comments written by agents discuss history unrelated to the current code.

The speaker particularly values one principle: the shortest path should also be the best path. Agents naturally seek shortcuts, so design the fastest path to be the best path.

Only after that come softer constraints like rule documents, skills, and bugable. He warns that if you rely only on soft constraints, the codebase will eventually become messy—it's only a matter of time. Technology stack choice therefore becomes important—Rust's strict compiler and borrow checker are examples of hard constraints.

useEffect is a hook in React used to handle side effects, often executing data requests, subscribing to events, or manipulating the DOM when a component mounts or dependencies change. It is flexible, but it can also easily lead to race conditions, duplicate requests, memory leaks, and hard-to-trace render loops—especially when agents generate large amounts of code, misuse of useEffect can quickly accumulate technical debt. Growth Bar directly prohibits useEffect at the architectural level, essentially elevating "don't use this hook" from a code convention to a hard failure at the compilation/check stage, forcing developers (and agents) to adopt more predictable data flow solutions, such as centralizing side effects in the routing layer or state manager.

Cost and Collaboration: Return on Investment Is the Key

Regarding the question of whether "ordinary-quota users can replicate this approach," the speaker acknowledges that he works at an AI lab where tokens are nearly unlimited, but he sees this first as a return-on-investment issue. The early investment of tokens in refactoring the codebase and building constraints pays off in efficiency gains for the entire team—designers and product managers can also submit high-quality code, even directly fix problems and then ask engineers to review.

He uses Growbot as an example: this new application gave non-technical colleagues their own "Cruiser moment," with an interface as natural as iMessage, where everyone can coordinate multiple agents with independent identities. Product managers are already accustomed to having agents summarize the work engineers completed the night before, and can also directly deliver features.

The cost is real. To refactor Growth Bar onto the new architecture, he made over 600 PRs; but precisely because of this early investment, he now almost never reads code line by line, CI strictly gates changes, and he doesn't have to worry at midnight about someone merging a performance regression.

Starting with validation, building trust locally, then expanding to cloud agents that automatically pick up tasks, and only finally allowing auto-merge—there are no shortcuts on this path, because it depends on each engineer's own level of trust and engineering quality standards.

Rust's borrow checker is part of the Rust compiler that statically verifies reference lifetimes and ownership rules at compile time, ensuring memory safety issues such as dangling pointers, data races, or use-after-free do not occur. For agents, this means many logic errors that would silently pass in JavaScript/Python directly cause compilation failures in Rust, and the agent must fix them before continuing. This "hard constraint" moves code quality assurance forward from human review to the compilation stage, reducing deep defects discovered only after CI. Technology stack choice is therefore not just a performance preference, but also a means of constraining agent behavior and reducing validation costs.