Why AI Coding Agents Fail: Breaking Down the Five Core Architectures of a 'Software Factory'

What Is a "Software Factory"?

Everyone talks about the "Software Factory," yet very few people define it clearly in code. At its core, it's simply a running loop: issues arrive from a backlog queue, a coding agent picks one up and writes a fix, an independent system validates the work, and a human decides whether to merge.

The most bare-bones version is a single prompt — you hand an issue to an agent, tell it to fix the bug, write tests, and open a PR. For a single ticket, this already works, and it genuinely qualifies as a micro software factory. The problem is speed: one agent handles one ticket at a time, so 10 issues means waiting through 10 sequential rounds.

The obvious upgrade is to spin up 10 agents in parallel. And that's precisely where everything starts to break.

Where Parallel Agents Go Wrong

The author demonstrates this with a real scenario: TinyLinks, a URL shortener that drops the https prefix and causes redirects to fail. Throwing multiple agents at it in parallel exposes a series of structural problems.

Shared folder conflicts: If all agents work inside the same local checkout directory, their changes pile up in the same place. In the author's test, three agents shared a single folder, resulting in commits that touched six files across three separate issues — a complete tangle. The fix is one isolated sandbox per agent, each with its own copy of the code running on its own machine.

Duplicate claiming: Two agents can grab the exact same issue at the same moment. The fix is a "claiming" mechanism: before starting work, an agent labels the GitHub issue claimed, and other agents skip anything already labeled.

Agents lying about test results: This is the most insidious problem. In the author's test runs, the worker agent reported "tests passing" every single time — even in a case where the independent verifier clearly saw failing tests. The fix is an independent verifier: a fresh sandbox that cold-checks out the working branch and runs tests itself, paired with a separate reviewer agent that inspects the diff line by line.

Stale APIs due to training cutoffs: Models only know the APIs they saw during training, and frameworks keep changing. The fix is giving the agent a real-time documentation lookup tool to query current docs before writing code.

PR flooding: These agents love opening PRs. The fix is that the factory is only permitted to open PRs — the merge button stays strictly in human hands.

---

The race condition in the claiming mechanism deserves extra attention. Labeling an issue is itself a network write operation, so in theory two agents could read an "unclaimed" state within an extremely narrow time window and both apply the label simultaneously. Common engineering mitigations include using GitHub API conditional requests (ETag/If-Match) for optimistic locking, or maintaining an atomically updated task scheduling table externally (e.g., Redis SET NX). For smaller-scale factories, tolerating occasional duplicate claims and letting the verifier stage catch them is usually sufficient — but as the number of parallel agents grows, idempotency in the scheduling layer becomes a critical bottleneck.

The Four Sandbox Primitives That Make the Factory Work

For a software factory to function, you need agents that can run for extended periods in the cloud. This example uses Upstash Box as the sandbox — think of it as a lightweight, on-demand cloud computer purpose-built for hosting agents. Each box comes pre-loaded with an independent filesystem, bash, git, node, and agent execution environments like Claude Code and Codex.

What actually makes the factory viable are four key primitives:

Fast Boot and Disk Snapshots

Calling create gives you a live Linux machine in about two and a half seconds. You don't want 10 parallel agents each doing a fresh git clone and npm install from scratch, so you install dependencies once on a base box and take a snapshot. That snapshot freezes the disk state into a reusable image — you can then boot five machines in parallel, each waking up with dependencies already in place.

Filesystem and Pause Isolation

Files written in box 1 are invisible in box 2 — each agent has its own private disk. Even better, when an agent is waiting for human input, you can pause the box, freezing its disk state and stopping compute billing. When it resumes, the files are exactly where they were.

Network Egress Firewall

If you're letting agents run untrusted code or install packages, you don't want them talking to random servers on the internet. Upstash lets you lock down network egress per box. For example, you can allow access only to the NPM registry, and any other connection attempt gets blocked.

True Remote Execution

All computation runs on cloud servers, not your local laptop. When you ask Claude Code about its runtime environment inside a box, it reports a remote Linux host with an IP completely different from your local MacBook — meaning agents keep working even after you close your laptop.

---

Disk snapshots aren't a new concept in the container and VM world, but using them for agent parallelization has distinct value. Traditional CI pipelines typically reinstall dependencies on every build — acceptable for a handful of serial tasks, but when you need to instantly scale out dozens of identical environments, cold-start time directly caps task throughput. Upstash Box snapshots are conceptually similar to cloud VM AMIs (Amazon Machine Images) or OCI image layers for containers, but at a finer granularity targeting disk state rather than image build steps. This allows delta-forking from a pre-installed-dependency baseline, avoiding the I/O overhead of each agent re-running npm install.

The Full Pipeline: From Claiming to Merge

The author's test subject, TinyLinks, had 10 open issues: 4 bugs, 5 new features, and 1 intentionally vague one — used to test whether the factory is smart enough to abandon a meaningless ticket. Scoring was done via one hidden test per issue, stored on the author's local machine outside the repository and completely invisible to the agents. At baseline, all 9 hidden tests on the main branch returned zero passes.

The pipeline breaks into distinct stages:

• Triage: A fresh box reads the issue and determines whether the requirements are clear enough to build. If not, the ticket is rejected.
• Worker: Once triage passes, a new box writes code, adds unit tests, and pushes a branch.
• Verifier: Never trust the worker agent's test report. A clean snapshot cold-checks out the branch and runs tests independently.
• Reviewer: Within the same verifier box, a reviewer agent inspects the git diff line by line, catching logical bugs that unit tests miss. Only when both tests and review pass does the factory open a PR.

Real-time documentation lookup is handled by Context7 — a single web request that searches official docs. The security design is elegant: the Context7 API key is attached by Upstash to outbound requests outside the box. Agents inside can read documentation but never see the key.

Why Both Verifier and Reviewer Are Non-Negotiable

The full run of 10 issues completed in four and a half minutes, with five parallel pipelines running at a time. The factory opened 9 PRs and skipped the vague issue as expected. When validated against the external test suite, all 9 branches passed the hidden tests.

The most valuable moment came from a PR that the reviewer rejected. In that run, the worker agent said tests passed, the independent verifier ran tests and they passed, and even the author's hidden evaluation tests passed — but the reviewer, reading the diff, realized that if two links were created within the same millisecond, the link list return order would be wrong, and reproduced the bug. The worker agent's unit tests missed it because the test data used pre-sorted slugs.

The conclusion is straightforward: passing tests doesn't mean the code is correct. A dedicated code review agent, separate from the worker, caught a sorting bug that all other tests missed.

The Overlooked Fifth Stage: Integrator

9 PRs looked perfect in isolation, but when merging into main, 7 of the 9 immediately threw merge conflicts. The reason: all agents started from the same base snapshot, and most added their own unit tests to the same test file — without any agent knowing the other eight existed.

The fix is a fifth stage: the Integrator. It's another box that pulls each reviewed PR one by one into a unified integration branch. Whenever git merge hits a conflict, the Claude Code agent inside opens the conflict markers, merges both sides to ensure nothing is lost, then re-runs the full test suite after each merge. In this case, all 9 PRs integrated cleanly, with 7 conflicts resolved automatically by the agent. The final step belongs to a human — one click of merge, and main correctly rejects links missing https.

---

Parallel branches producing conflicts on merge comes down to the fact that each agent's work is based on the same common ancestor commit — what version control theory calls the "diamond merge problem." Each branch evolves independently, and when they both modify the same region (like the end of a test file), Git can't automatically determine the final state and must insert conflict markers for human (or agent) resolution. Traditional strategies fall into two camps: rebase, which places each branch sequentially after the previous one, maintaining linear history but requiring frequent commit rewrites; and sequential merge — the approach the Integrator uses here — merging one by one and re-running tests at each step to minimize the risk of introducing breaking changes.

Cost and Key Takeaways

The cost math is quite favorable: running the factory across 10 issues and four stages consumed approximately $3.67 in Claude Code token usage; the integration stage resolving 7 conflicts and repeatedly running tests cost approximately $6.37 in token usage. Upstash compute box usage for an entire day (including all test runs) was only about $0.10 — because coding agents spend the vast majority of their time waiting for LLM API token streams, and Upstash only bills for active CPU time, making sandbox compute nearly negligible.

Two things to keep in mind: First, sandboxes prevent agents from stepping on each other's files, but they don't make the code correct — code quality still comes from your tests, verifier, and reviewer. Second, parallel agents working on shared files (test suites, routers) will create conflicts, and you must plan for an integration step.

At the end of the day, a software factory is just a loop: issues come in, agents build fixes, independent boxes validate, humans merge. Every piece of the architecture — one snapshot box per agent, issue claiming locks, independent verifiers, real-time documentation lookup, a conflict-resolving integrator, and a human with the final say — was added precisely because parallel agents break without each of them.