Software Factory in Practice: Building an Evaluable Self-Evolving Development Pipeline with Codex and Claude Code

Title

Software Factory in Practice: Building an Evaluable Self-Evolving Development Pipeline with Codex and Claude Code

Meta Description

Deconstructing a software factory architecture built on Codex and Claude Code: control plane and data plane separation, foreman prompts ensuring consistency, triggers automatically pulling tasks, and how to use runtime data and evaluations to continuously optimize the agent coding workflow.

Article Content

When models, prompts, execution harnesses, or skill sets change frequently, a practical problem faces every developer who heavily uses coding agents: you cannot tell how much actual benefit these changes bring. The "software factory" solution shared on Bilibili is valuable not because it replaces local coding agents, but because it provides a repeatable, measurable, and continuously improvable engineering process.

According to the video's author, Owain Lewis, he works both as an independent developer and as an engineer at a large enterprise, so the design accounts for both solo and team scenarios.

What Is a Software Factory: Turning Agent Coding into an Observable Pipeline

The author defines a software factory as: an underlying infrastructure that lets coding agents run according to a defined process. You put in work, and the system executes it through exactly the same process every time, recording all metrics and data from that run.

This solves several typical pain points in agent coding:

• No more manual management of worktrees, branches, and local environment configuration
• Once a task is delegated, the system handles queuing, execution, and result collection
• Every run leaves a historical record for longitudinal comparison

He solved the conflict between security and access using SSH tunnels: both the control plane and worker nodes run on VMs with no public internet entry point, and he connects locally through a tunnel, maintaining privacy without needing a complex security architecture. This is especially friendly for independent developers—no extra architectural cost for security.

worktree is a Git multi-worktree feature: the same repository can check out multiple independent working directories, each corresponding to a different branch, sharing the same object database. For coding agents, executing each task in an independent worktree avoids interference from branch switching and uncommitted changes, and if a task fails, the entire directory can simply be discarded. The software factory automates this process, so developers no longer need to manually create, switch, and clean up worktrees—this is one concrete manifestation of how it turns "local manual operations" into a "systematic pipeline."

Architecture Breakdown: Control Plane and Data Plane

The entire system is divided into two layers, and this division is the foundation for understanding all subsequent design.

Control Plane: the Scheduling and Observation Center for Work

The control plane handles task management. After tasks are submitted, they are queued; when completed, results flow back to the control plane, forming a queryable dashboard. From the video demonstration, the UI shows lists of completed and in-progress tasks, as well as analytics such as average task duration (18 minutes in the example) and cumulative token consumption (15 million in the example).

Data Plane: the Division of Labor Between Workers and Executors

The data plane consists of Workers. Here is a distinction that is often confused: a Worker is not the same as an agent executor. The Worker is responsible for pulling tasks, setting up the environment, handing the task to the corresponding coding agent, and then recording results back to the control plane. Workers can scale horizontally—multiple VMs, Kubernetes Jobs, or even cloud instance pools.

The executor layer supports different models such as Codex and Claude, with the model list explicitly declared in configuration. This means you can run the same task on different executors and let the data speak.

Kubernetes Job is a controller in K8s for running one-off tasks: it creates one or more Pods, considers the task complete once the specified number of Pods terminate successfully, and retries according to policy on failure. Jobs are naturally suited to the software factory's Worker scenario—each task spins up a Pod to execute, automatically reclaimed when finished, scaling on demand. Compared with long-running VMs, Jobs offer finer scheduling granularity and higher resource utilization, but require cluster infrastructure; the author lists them as one of the optional scaling methods for Workers, which precisely shows that the Worker layer is deliberately designed to be decoupled from executors and replaceable by environment.

The Foreman Pattern: Ensuring Consistency with Unified Prompts

Every task in the system is first handed to a "foreman" coordinating agent. The foreman breaks down the task and dispatches subtasks to sub-agents in a fixed order, until code changes are completed and labels are applied.

The author emphasizes that this process even encodes the label management approach into configuration. When you modify the foreman prompt, every subsequent task uses the new version—this is the source of "consistency." Conversely, if each developer uses different prompts and skill sets, the team loses any basis for comparison.

The system also configures two types of triggers: one periodically scans the codebase, using a cheaper model to perform health checks, find defects, and automatically open tickets; the other listens for specific labels on GitHub and automatically pulls work into the factory as soon as they appear. These are the two entry points for work entering the system.

Evaluation System: Making Every Change Evidence-Based

This part is the most valuable part of the entire solution, and also the author's starting point.

In factory mode, all tasks follow exactly the same sequence of steps, so when you want to switch models or adjust prompts, you can run for a few days, accumulate enough data, and then compare changes in cycle time and task completion rate. This is a low-cost controlled experiment.

The author also mentions a set of "semi-deterministic" evaluations: through automated checks and tests, verifying whether labels are correct and whether the workflow is followed. In addition, every run produces structured metadata, including duration, token count, number of command executions, agent turns, etc., which can be fed directly to an agent for interpretation.

A concrete case illustrates the point well: he had an agent log into a VM via SSH, review the outputs of recent runs, and look for prompt optimization opportunities to reduce token consumption. The agent's recommendation after analysis was to change some steps from "agent-driven" to deterministic Python scripts, with scripts monitoring CI and then delegating to the agent. Scripts are cheaper, more predictable, easier to test, and produce more stable output structures.

This improvement re-enters the factory by applying a label, is implemented by the agent itself, and then the effect is verified with data. If validated, the same change is ported back to the local development environment. The author points out that because he reuses the same coding loop on every task every day, the benefits of such improvements are amplified across all work.

This closed loop of "analyze runtime data → discover improvement points → automate implementation → data validation" has more long-term value than simply pursuing stronger models.

Deployment and Cost: Practical Considerations for VM Selection

The author chose to deploy both the control plane and Workers in the cloud, reasoning that he wanted to separate the local development environment from agent infrastructure, and that it also helps him think about the architecture more clearly.

On cost, he gives a concrete comparison: an E2 medium in Google Cloud's London region costs about $32/month (excluding disk), and comparable configurations on Google Cloud also approach $30/month; while Hetzner's Ubuntu 16GB instance (Helsinki) is significantly cheaper. His own approach is—put the bulk of infrastructure on Google Cloud, but choose more affordable providers for cost-sensitive parts. Of course, if the budget is limited, running locally is entirely feasible.

The database used is SQLite, with configuration items including the maximum number of concurrent tasks, prompt definitions for the foreman and audit tasks, and the list of executors and available models. The author emphasizes that the executor layer is extensible—you can plug in any coding executor for comparative experiments.

Team Scenarios: the Leverage Brought by Shared Infrastructure

The author has worked in large enterprises with hundreds to thousands of developers, and has a clear judgment on team scenarios: shared infrastructure lets everyone use the same workflow, and any improvement to prompts or configuration benefits the entire team.

By contrast, if each developer uses different models, different harnesses, different prompts, and skills, there is neither consistency nor repeatability at the team level, and improvement becomes almost impossible to discuss. He compares this to CI/CD: system-level improvements bring collective leverage, rather than scattered accumulation of individual experience.

Applicability Boundaries: Not All Work Suits the Factory

The author's attitude is quite restrained. Design work and tasks requiring repeated iterative exploration are not suitable for the factory—such work needs to stay close to the code, involve rapid back-and-forth, and use browser verification. He himself still uses coding agents locally for this kind of work.

The typical scenario for the factory is: queue up a large number of tasks, schedule them for batch execution overnight, advance them in order, and review the results together the next morning. In other words, local agents and the software factory are complementary, not substitutes.

For developers who want to improve the determinism of agent coding, this solution offers a clear lesson: first establish observable run records, then use data to drive process improvement, and finally solidify validated changes into the process. Tool choice is secondary.

One-line Summary

The software factory solution shared on Bilibili uses an observable pipeline to make agent coding repeatable and measurable.

Paragraph Summary

The article introduces an agent coding infrastructure solution called the "software factory," whose core value is not replacing local agents but providing a repeatable, measurable engineering process. The system is divided into a control plane and a data plane: the control plane handles task queuing, result collection, and dashboard observation; the data plane consists of Workers, which pull tasks, set up environments, and hand them to executors such as Codex and Claude—the two concepts must be distinguished. Each task is first handed to a unified "foreman" agent for decomposition and dispatch, ensuring consistency in process and label management. The most valuable part of the solution is the evaluation system: all tasks follow the same steps, so you can run for several days and then compare cycle time and completion rate, and use structured metadata for controlled experiments. In one case, an agent analyzed runtime data and recommended changing some steps to deterministic Python scripts to reduce token consumption, forming an "analyze—improve—validate" loop. The article also discusses private deployment via SSH tunnels, VM cost comparisons, and the leverage effect of team shared infrastructure, and makes clear that design-iteration work still suits local agents—the two are complementary.