The Five Levels of a Software Factory: A Progressive Path to AI-Automated Programming

What Is a Software Factory

A software factory is a set of repeatable processes: clear requirements go in, pass through a series of checkpoints along the way, and tested software comes out the other end — with a complete audit trail. Johnson, a software architect with 15 years of technical experience and over 8,000 hours of AI-building practice, emphasizes in his analysis that writing code is just one step in the process, not the whole thing.

Every level answers the same question: What can the system now handle without your involvement? Higher levels don't mean more agents or more powerful models — they mean you've handed off yet another piece of the work. And every time you hand something off, there must be another mechanism to check the quality of that work.

A critical point: no matter how high you climb, the figure representing "you" is always present in the diagram. You move upward as the levels rise, but you never disappear.

To make this concrete, Johnson uses an online store as a running example throughout — it has products, a shopping cart, and a checkout page. A ticket comes in: allow customers to use discount codes at checkout. Sounds simple, but there's one bug that must never reach production: a customer stacks two discount codes and the total goes negative — meaning the store would end up paying people to shop.

Level 1: Context (Context Engineering)

A coding agent knows how to write code right out of the box. What it doesn't know is your project. Context engineering is how you bridge that gap — you give it your project's rules, the commands to run tests, and the tools it's allowed to use. For the discount code ticket, that means pricing rules, test commands, and the most critical business rule: the total price can never go below zero.

Context can extend to "skill packs" like Obra's Super Powers or GitHub's Spec Kit, which give agents a structured way of working — how to plan, how to write tests first, how to review their own changes.

An important caveat: more context doesn't automatically mean better. Research testing context files like Agent.md across multiple coding agents found these files typically didn't improve success rates — they just drove costs up by more than 20%. So keep it short, and only tell the agent things it couldn't reasonably guess.

At Level 1, the agent does the building, but you still make every decision — reading diffs, testing, telling it what to fix, and telling it what to build next.

Level 2: Loop (Loop Engineering)

At Level 1, you're the one saying "fix that" after every attempt. Level 2 hands that job off. You give the agent a goal (go) and a set of checks, then it builds, runs the checks, reads the failures, fixes them, runs again — looping until everything passes.

Two things can go wrong inside the loop. First, the agent might think it's done before it actually is. So "done" must be determined by the checks — and the agent must never be allowed to modify the tests to make them pass. Second, if you haven't set the right criteria, it might run forever, burning through tokens. So the loop needs a sensible upper bound — stop after a certain number of attempts and report that it's stuck.

Even when all tests pass, it's not over. Meter analyzed AI pull requests that passed tests and found that roughly half wouldn't be merged by the project's maintainers. So passing tests means ready for review, not ready for production.

At Level 2, you've handed off the correction loop, but you still own the goal and the final review. You're now reviewing once at the end rather than watching every step.

---

The "checks" mentioned here typically span multiple layers in engineering practice: unit tests verify the logic of individual functions, integration tests verify how modules work together, and static analysis tools (linters) check code style and potential errors. The agent relies on feedback from these checks inside the loop to determine whether its changes are working. This is exactly why "the agent must never modify the tests" is so critical — if allowed to, the agent might take the path of least resistance: deleting or altering failing tests to make them pass, rather than actually fixing the problem. This behavior is known as reward hacking in reinforcement learning, and it's one of the most important failure modes to guard against in automated loops.

Level 3: Queue

The loop in Level 2 runs automatically, but you still have to open a terminal or cloud environment, enter the goal, and wait — one task at a time. Level 3 hands off the act of "starting" as well. Work comes in as tickets; when a ticket is ready, the system picks it up automatically, runs in the background, and comes back with a completed pull request.

Johnson demonstrates this with his open-source project Cloud Code Symphony (inspired by OpenAI's Symphony). It watches a Linear board, and when a ticket moves to "spec ready" status, it assigns that ticket its own isolated copy of the repository and a dedicated Claude Code agent. Managed background agents like Devin AI work similarly.

In the demo, a ticket moves from "spec ready" through "agent active" to "needs review" in under a minute, with Johnson not typing a single command. The agent wrote 14 tests covering every case in the ticket — all passing — and independently made one decision he never asked for: "free shipping is calculated after the discount." That might be right or wrong, which is exactly why the PR waits for his approval.

But the hard part of a queue is handling all of this correctly: not starting the same ticket twice; retrying on failure; keeping a log of every run; and the final check still belongs to you.

Tharo's research on over ten thousand developers using heavy AI tooling found that people were merging 98% more pull requests — but review time had also risen by 91%. The queue didn't eliminate the bottleneck; it moved it onto you. The queue never gets tired. You do. That's the failure point of this level.

---

The "isolated copy of the repository" mentioned here is typically implemented via Git branches or sandbox environments, ensuring that multiple agents working on different tickets in parallel don't interfere with each other. This is the foundational requirement for a queue system to run concurrently — each ticket gets its own isolated workspace, preventing race conditions where two agents modifying the same file simultaneously cause code conflicts. Audit logs are another critical piece of infrastructure, recording the inputs, outputs, duration, and status of every run so that failures are traceable and problems are reproducible. These two infrastructure elements — isolated environments and audit logs — are what separate a queue that "sometimes works" from one that "works reliably."

Level 4: Orchestration

A queue only runs the ticket you give it — nothing more. But a complete discount code feature is actually three interdependent pieces: the pricing rules, the input field on the checkout page, and the order summary displaying the discount. Someone has to break it apart and stitch it back together — and at Level 3, that someone is still you.

Level 4 adds a "lead agent" on top of the queue, responsible for orchestrating — deciding which tickets exist and when each one is ready. Johnson demonstrates this using the T3 code framework, locking down a test file that covers the entire purchase flow. The agent can run it, but cannot modify it.

The reason this level exists is precisely to handle a specific class of failure: the pricing side sends discounts in cents, while the checkout side's tests are still written in dollars and the agent never updated them. Every individual ticket's tests pass — pricing passes, checkout passes, order summary passes — but when the locked end-to-end test runs the complete purchase flow, a $45 order minus $20 produces the wrong negative total. Every ticket passed. The feature is still broken. That's why the end-to-end check must cover the entire feature, not just each individual task — and why it must be locked before work begins.

At Level 4, you've handed off the coordination work, but you still own product decisions and the release. You're now looking at a finished feature, not a pile of pull requests.

Level 5: Dark Factory

To hand off the release as well, the checks must be good enough to stand in for your judgment. So how do you know the checks are good enough? You try to break your own feature. The bugs that really hurt are the ones nobody thought to test for.

Before handing off the release, there are three tasks that are easy to conflate. First, verifying that the change itself is correct. Second, checking the checks — feed it a checkout case with incorrect charges or stacked discounts and see whether it catches them (this is evaluating the evaluators, i.e., evals). Third, setting the rules — what the system is allowed to deploy, and what still requires you. Even perfect checks can't choose which risks you're willing to accept.

Johnson stresses that another agent saying "looks good" is not evidence. Evidence is the bad changes your checks caught, and the record proving good changes still pass — that's your scorecard. Every time you swap out a model or change the workflow, you have to run it again.

At Level 4 you approve changes; at Level 5 you approve the rules under which changes can go live. For the class of changes you've pre-approved, the dark factory system builds, checks, and deploys with no human review.

But a fully autonomous dark factory doesn't yet exist. Johnson searched for production-level cases confirmed by independent third parties and found none. The closest example is Chainguard: 840 pull requests merged over eight weeks with no human intervention. But a closer look reveals it only targets one type of change — fixes bringing code back to their own standards; AI scoring alone isn't enough to merge, a set of rule-based checks must also agree; and anything touching production infrastructure or authentication goes to a human every time. What they handed off was only the "merge" step.

So Johnson doesn't consider this a dark factory. He calls it a "dark room": a room with the lights off, while the rest of the building still has people on duty. When someone says they're running a dark factory, ask two questions: dark for which class of changes? And dark up to which step?

---

The "evals" mentioned throughout refer to a set of automated mechanisms for evaluating the quality of an AI system's outputs — essentially "tests that test the tests." In traditional software testing, we test whether code works as expected; evals go a step further and test whether the standard for judging code correctness is itself reliable. The approach is to build a set of "golden samples" with known answers: changes known to be defective (expected to be blocked) and changes known to be correct (expected to pass), then run the checking system over them and measure its accuracy. An evals score is the basis for trusting your checking system — and the quantitative evidence that determines whether you can move to a higher level of automation. Without evals, "the checks are good enough" is just a feeling, not evidence.

Find Your Level with Five Questions

Johnson offers five yes/no questions — but answer them for a specific type of work you do, not your entire company:

• Does your agent know the project rules without you re-explaining them every session?
• When a check fails, does the agent automatically fix it and retry without you telling it to?
• Does ready work start without you manually triggering it?
• When the pieces don't fit together, can the system detect and fix that on its own?
• Is there a class of changes that can go live without your approval?

Find your first "no" — that's the direction to work on next. If it's the first question, write down your project rules and keep them short. Second question: give the agent a test it can't modify and a retry limit. Third question: let tickets kick off work and keep an audit log of every run. Fourth question: write the end-to-end feature check before splitting up the work. Fifth question: don't rush — first verify that your checks and evals are solid enough.

Two important reminders: higher isn't always better. If work is rare, goals are ambiguous, or the cost of mistakes is high, staying at a lower level may actually be the right call. And these levels aren't a strict sequence — they're a common path, not a rule.

Notice that most of this work is about "checks." You build the checks first, then you hand off the work. More responsibility requires stronger evidence, and evals are how you get that evidence — that's the one-sentence summary of the whole framework.