[KongchangAI]
· 5 min read· 2,680 words

Deconstructing Codex Harness Engineering: The Secret to a Million Lines of Code with Zero Handwriting

Deconstructing Codex Harness Engineering: The Secret to a Million Lines of Code with Zero Handwriting

OpenAI Codex produced 1M lines of code with zero handwriting in 5 months, driven by environment construction and engineering habits.

Based on Bilibili's "Harness Engineering Five Lectures" part 4, this article distills OpenAI Codex's method for producing ~1M lines of code with zero handwriting in 5 months. Key insights: agent speed depends on environment clarity, not model capability; agents.md is a ~100-line map, knowledge lives in repo docs; code checks enforce invariants with fix-oriented errors; golden atom rules and background garbage collection control drift. Prompt caching and compaction optimize the loop. Engineering habits—repo as single source of truth, invariant checks, garbage collection—matter more than tools.

OpenAI's Codex team once ran an experiment: over 5 months, without writing a single line of code by hand, they relied entirely on AI agents to build an internal product, eventually reaching roughly 1 million lines of code in the repository. This experiment popularized the term "Harness Engineering." Recently, a Bilibili creator deconstructed Codex and OpenAI's engineering approach in detail in the fourth installment of the "Harness Engineering Five Lectures" series. This article is based on that video content, combined with multiple technical sources, to distill the core logic.

Experiment Background: 5 Months, 1 Million Lines, Zero Handwriting

According to an article by OpenAI engineers Ryerson and Ian Lopopolo, the experiment began with an empty repository. The very first commit was generated by Codex, including the directory structure, CI configuration, formatting rules, and the agents.md file that guides agents' work. Five months later, the repository had accumulated approximately 1 million lines of code and merged around 1,500 PRs. The team initially had only 3 engineers driving Codex, averaging 3.5 PRs per person per day; later this grew to 7 people, and the per-capita speed actually increased. The product had several hundred internal users using it daily, and was also launched, deployed, and fixed. The team estimates that handwritten code accounted for only about 1% of the time, with everything else generated by agents—covering product code, tests, CI, release tooling, internal tools, documentation, review comments, and even scripts for managing the repository itself.

The slogan of the experiment was "humans grow in number, agents execute." But initial progress was slower than expected—not because Codex lacked capability, but because the environment wasn't clearly specified. Agents lacked tools, abstractions, and internal structure. So engineers' primary work shifted from writing code to enabling agents to do the job well. Each time they hit a problem, they asked "what capability is missing" rather than "let's try harder."

Instruction Design: Give a Map, Not a Manual

The team's earliest lesson: give Codex a map, not a thousand-page manual. They once tried stuffing all rules into one large agents.md file, but it failed for several reasons: a huge instruction file eats up context, causing agents to miss key constraints or fixate on the wrong ones; when everything is important, nothing is; manuals quickly go stale and become a graveyard of rules; and there's no way to mechanically check coverage and link validity.

So agents.md was designed as a table of contents—only about 100 lines—serving as a map telling agents where deeper knowledge lives. The real knowledge is placed in the repository's structured documentation directory, covering architecture descriptions, design docs, product specs, quality scores, security and reliability requirements, and execution plans. Plans are first-class citizens: small changes use lightweight ad-hoc plans, while complex work is written as execution plans with progress and decision records, committed directly into the repository.Committed directly into the repository

For agents, anything not available in context at runtime effectively doesn't exist. The team worked to push more and more context into the repository—for example, through dedicated inspection tasks to confirm documentation freshness and link correctness, and periodically running a documentation gardener agent to fix outdated docs. When Codex reads agents.md, it first reads the global file, then goes layer by layer down from the project root, taking at most one file per layer, with closer files taking priority, and a total token limit of 32k. Compared to the previous lecture's Claude Code recommendation of under 200 lines, the Codex team kept it around 100 lines—same philosophy: instructions should be short, layered, and local.

Environment Construction: Make the App Visible to Agents

As code output accelerated, the bottleneck shifted to manual testing capacity. The team made the application itself visible to Codex: first, every change can spin up its own instance that Codex can operate; second, they connected the browser debugging protocol to the agent, providing page snapshots, screenshots, navigation, and other capabilities so it can reproduce bugs and verify fixes; third, they exposed logs, metrics, and call traces through local monitoring tools, so agents can write their own queries to verify performance requirements (e.g., service startup within 800 milliseconds, critical paths within two seconds).

These capabilities let Codex work continuously for over six hours—often while humans slept.Codex's two hands

The browser debugging protocol (typically referring to Chrome DevTools Protocol, CDP) is a standardized interface that lets external programs control the browser. It allows agents to get DOM snapshots, take page screenshots, click elements, read console logs, monitor network requests, and even execute JavaScript. For agents that need to reproduce UI bugs or verify front-end fixes, this is equivalent to giving them a pair of hands that can "see" and "operate" the page. It relies on the same underlying technology as traditional automation testing frameworks (like Playwright, Puppeteer), but agents can dynamically decide the next action rather than executing preset scripts. After connecting CDP, Codex can open pages on its own, locate anomalies, and verify fix results—automating steps that previously required manual clicking and confirmation. This is one of the key supports enabling it to work continuously for over six hours.

Constraint Mechanisms: Hold the Invariants

Documentation alone can't govern a codebase generated entirely by agents. The team's principle is to hold the invariants—for example, requiring Codex to parse and validate data at boundaries, but not specifying the method. Architecture is rigidly defined: each business domain has fixed layers, dependencies can only flow forward; cross-cutting concerns (like feature flags) can only enter through a unified entry point. These rules are enforced through custom code checks and structural tests—and the checks themselves are written by Codex.

A clever detail: check error messages are written as fix instructions. When an agent violates a rule, it doesn't just see the error—it sees how it should be fixed. These rules might seem nitpicky for humans, but for agents they're statements—write once, apply everywhere. Human taste is continuously fed into the system through review comments, refactoring, and user feedback; whatever documentation can't govern gets escalated to code checks.

Once speed picked up, traditional engineering rules became counterproductive. The repository has almost no blocking merge gates; PRs are short-lived; occasionally failing tests are mostly just rerun—because when agent output far exceeds human attention, fixing mistakes is cheap and waiting is expensive. But complete hands-off would let the codebase drift. The team once spent 20% of time every Friday cleaning up AI-generated garbage code, then shifted to writing "golden atom" rules into the repository (e.g., prefer shared toolkits, avoid duplicate small functions, boundary validation or typed interfaces), periodically running background Codex tasks to scan for deviations, update quality scores, and open refactoring PRs—most reviewed and auto-merged within a minute. This is called garbage collection. Technical debt is like a high-interest loan: continuous small repayments beat paying it all off at once.

"Invariant" is a core concept in software engineering, referring to conditions that must always hold true during system operation or code evolution. It can manifest as data structure constraints (e.g., order amounts can't be negative), architectural rules (e.g., the domain layer can't depend on the infrastructure layer), security requirements (e.g., all external input must be validated), or performance baselines (e.g., upper limits on critical API response times). "Holding the invariants" throughout this article means: no matter how freely agents implement details, these conditions cannot be broken. Writing invariants as executable code checks or structural tests is equivalent to solidifying human design intent into machine-verifiable rules, so agents can be automatically blocked on violations at every commit. This approach upgrades "review comments" from one-time human feedback to permanently effective automated constraints—the infrastructure for managing large-scale agent-generated codebases.

The Codex Loop: Prompt Caching and Compaction

An article by OpenAI engineer Michael Bolin reveals details of the Codex loop. Each model call includes base instructions, a tool list, and input messages, with several fixed messages inserted at the front: one explaining sandbox rules, one loading agents.md and a skills overview, and one describing the current environment. Sandbox rules only govern Codex's own command-line tools; tools connected via MCP are not covered and must handle their own security.Not under his kill authority

Each tool call result is appended to the request, and data volume grows quadratically. But prompt caching lets computation be reused directly when the old request's prefix matches the previous one exactly, turning cost from quadratic to linear. So Codex deliberately ensures the old request is a prefix of the new request—avoiding mid-stream changes to tool lists, model, sandbox settings, or working directory. When connecting MCP, cache misses once occurred because tool order wasn't fixed; the later rule was that configuration changes append new messages rather than modifying old ones.

When context fills up, Codex uses compaction instead of truncation: beyond a threshold, long inputs are replaced with entries representing the conversation, and OpenAI's API compaction feature returns encrypted content that preserves the model's understanding—humans can't read it, but the model can. This aligns with the ARC-AGI-3 benchmark setting of preserving reasoning rather than discarding it, where scores rose from 13.3% to 38.3% and output tokens dropped sixfold.

The sandbox defines boundaries; approval decides when humans must be asked. The sandbox governs all commands, using different tools on Mac and Linux, with three modes: read-only, working directory read-write, and fully open; two approval modes: ask on boundary crossing or never ask. There are also rules that decide allow/ask/forbid based on command prefixes, plus automatic review that sends approval requests to a review agent. Similar in spirit to Claude Code's background classifier.

Sub-agents reduce context pollution: the main agent handles requirements and decisions, while exploration, testing, and analysis go to sub-agents in parallel, returning only summaries. There are three built-in sub-agents—general-purpose, hands-on implementation, and code reading—customizable but with permissions inherited from the main agent. Read-heavy work suits parallelism; write-heavy work in parallel requires care about conflicts.

The hook system supports over a dozen events. The Stop hook is most worth examining: at the end of a round, if a reason is added to the return field, Codex won't refuse—it treats the reason as a new user message and continues. The official example is rerunning failed tests, with input tagged to avoid infinite loops. This is almost identical to Claude Code's Stop hook.Attach a reason

Prompt caching is an optimization technique in large model inference: when the prefix content of multiple requests is identical, the server can reuse previously computed key-value cache (KV cache), skipping redundant attention computation, thereby reducing latency and cost. For the agent loop, each round's request contains system instructions, tool list, history messages, and new tool results—the first half is usually unchanged. If computed from scratch each time, cost grows quadratically with conversation rounds; with caching, the old request becomes the new request's prefix, and incremental computation only covers new content, reducing cost to linear. The tradeoff is requiring a strictly stable request prefix—once tool lists, model parameters, or sandbox settings are modified mid-stream, the cache is invalidated. So Codex's engineering rule is "append, don't modify," which is both a performance design and a constraint on agent runtime configuration management.

Platformization and Comparison of the Three

Codex has a desktop app, command line, editor plugin, and web version, all sharing the same Harness underneath. The command-line version has been open source since last April (written in Rust); the plugin and cloud versions are not open source. Connecting the entry points is the App Server—originating from the editor plugin development when they didn't want to rewrite the loop and found MCP services didn't fit well, so they built a custom bidirectional protocol, now a stable platform interface. The protocol has three primitives: items (minimal input/output, like messages, command execution, code changes), turns (work caused by one user input), and ready-made (complete sessions that can be resumed, branched, and archived). Bidirectional means when the agent needs approval, the server can query the client and pause the turn.

Last August, OpenAI published an article positioning the open-source Harness as a platform, with the view that rather than having teams move their work into a generic chat box, it's better to bring agents into software designed around real work—the interface itself is the most important context. In the example, an agent sits next to a freight dashboard; the user selects cargo and clicks to compare recovery options, but consequential operations like changing cargo must be human-approved.

The course scores Codex's five subsystems: instructions are textbook-level (give a map, not a manual), tools range from command line to plugin products all self-selected, environment isolation is the signature, state relies more on repository conventions, and the feedback path is worth copying by default. Comparing the three: the Pi kernel is minimal and decides nothing for you, Claude Code has a complete kernel that works out of the box, and Codex open-sources the loop and writes rules into the repository. The UP host observes that Claude Code and Codex are growing closer—one adding, one subtracting—but now Codex also has hooks, sub-agents, skills, memory, and a review agent, with the Stop hook nearly identical. What truly sets them apart isn't the feature list but the engineering habits from OpenAI's article: the repository as the single source of truth, invariants written as checks, and regular garbage collection. These don't depend on the tool—you can do the same with Claude Code.

MCP (Model Context Protocol) is an open protocol for connecting large model applications to external tools and data sources in a standardized way. It defines a common interface for tool discovery, invocation, and result return, so agents don't need to write adapter code for each tool. The article mentions that after Codex connects MCP tools, these tools are not constrained by Codex's sandbox rules and must handle their own security; also, cache misses once occurred because MCP tool order wasn't fixed. Understanding MCP helps grasp the evolution direction of the current agent ecosystem: beyond model capability, the standardization of tool integration directly affects an agent's extensibility and security. Codex built its own App Server rather than directly adopting MCP precisely because MCP doesn't fully fit its loop needs for bidirectional interaction and session recovery—showing that tension still exists between standardized protocols and product-specific requirements.

Key Takeaways

  • The experiment proves zero handwriting is feasible: 1 million lines in 5 months, handwriting accounting for only 1% of time—the key is environment construction, not model capability.
  • Instructions should be short and layered: agents.md of about 100 lines serves as a map, knowledge goes into structured repository docs, and outdated content is cleaned regularly.
  • Constraints rely on code checks: hold the invariants, make errors into fix instructions, golden atom rules plus background garbage collection control codebase drift.
  • Loop optimization relies on caching and compaction: prompt caching ensures prefix stability, compaction replaces truncation to preserve reasoning, ARC-AGI-3 scores nearly triple.
  • Engineering habits matter more than tools: repository as single source of truth, invariant checks, regular garbage collection—applicable to any agent.

One-line Summary

OpenAI's Codex experiment produced a million lines of code with zero handwriting over 5 months, with the core being environment construction and engineering habits.

Paragraph Summary

Based on the fourth installment of Bilibili's "Harness Engineering Five Lectures," this article distills OpenAI Codex team's experimental method of producing roughly 1 million lines of code with zero handwriting over 5 months. The core argument: agent output speed doesn't depend on model capability but on whether the environment is clearly specified. The team designed agents.md as a ~100-line map rather than a thousand-page manual, put knowledge into structured repository docs; made the app visible to agents via the browser debugging protocol and local monitoring; held invariants through custom code checks and structural tests with error messages written as fix instructions; and controlled codebase drift with "golden atom" rules and regular background garbage collection. The Codex loop uses prompt caching for prefix stability and compaction instead of truncation, with sandbox and approval mechanisms converging with Claude Code. The ultimate conclusion: engineering habits like repository as single source of truth, invariant checks, and regular garbage collection set teams apart more than the tools themselves.

#Codex Harness Engineering#OpenAI Codex#AI agent coding#prompt caching#garbage collection#Claude Code
Share:

Related articles