The Design Secrets of 12 Production-Grade Agent Teams: Two Questions Decide Success

Why Everything Beyond the Model Matters More

All teams are using the same handful of models, yet their performance in production varies wildly. A researcher on Reddit read the publicly available material from 12 teams running agents in production — including Codex, Claude Code, Cursor, Devin, Stripe's Minions, Manus, Agentforce, Shopify Sidekick, Sierra, Intercom Fin, Harvey, and Hebbia — to answer one question: what is the layer of code around the model — the harness — actually doing?

According to Fortune's reporting on Anthropic's incident postmortem, Claude Code went through about a month of noticeable capability degradation in March and April: one change caused the model to discard its own reasoning history mid-session, and another limited replies between tool calls to 25 words or fewer. Throughout the whole process, the API itself did not change. The same model, a different harness, a completely different result. This is enough to show that the harness is not a nice-to-have, but the critical layer that determines whether an agent can work reliably.

Harness is the runtime framework built around a large language model, responsible for managing context injection, tool-call orchestration, state maintenance, error handling, and validation. It determines what information the model sees at each step, which actions it can take, and how its output is checked and corrected. In an agent system, the harness is essentially the model's "operating system": the model provides reasoning capability, while the harness determines how that capability is organized, constrained, and amplified. In the Anthropic incident, the API did not change but harness changes caused the capability degradation, which precisely shows the dominant influence of the harness on final performance. The industry is gradually reaching a consensus: the model is a commodity, and the harness is the source of product differentiation.

Three Consensus Points Across Twelve Teams

Despite differing product forms, these 12 teams show a highly consistent set of three practices in harness design.

Load Context Later

Codex maintains an AGENTS.md of about 100 lines as an index pointing to deeper documentation; Cursor writes long outputs to files and lets the agent read them when needed; Sierra uses progressive disclosure. No one defends giant static prompts anymore. Context is not stuffed in all at once but unfolded on demand, which both saves tokens and prevents the model from getting lost in irrelevant information.

Put Known Steps into Code

Stripe's blueprints mix deterministic nodes with agent nodes; Salesforce writes paths directly in code; even the most "agent-first" Harvey keeps document state firmly outside the model. Any process that can be predetermined is not left to the model's improvisation.

Check with Something Other Than the Generator

Devin's reviewer runs in a clean context; Harvey's drafting agent uses a diff tool to check its own edits; Shopify calibrated the agreement between its LLM judge and human judgment from kappa 0.02 to 0.61 (human-to-human agreement is 0.69). Separating generation from verification is the basic discipline that prevents errors from accumulating.

Cohen's kappa is a statistic that measures the degree of agreement between two raters, after subtracting the probability of chance agreement. Values typically range from -1 to 1, where 0 means no better than random guessing and 1 means perfect agreement. In the LLM judge setting, kappa is used to quantify how well automated evaluation matches human judgment. Shopify calibrating kappa from 0.02 to 0.61 means the judge went from nearly random to highly consistent with humans; human-to-human agreement of 0.69 serves as a reference ceiling. This metric is more convincing than simple accuracy because it rules out cases where raters happened to guess correctly, and it genuinely reflects whether an automated judge is reliable enough to independently take on verification duties.

Two Questions Explain Almost All Divergence

Beyond the consensus, the practices of the 12 teams diverge noticeably. The researcher found that almost all divergence can be explained by two questions: How cheap is it to discover an error? How costly is it to miss one?

Stripe is the clearest case. More than $1 trillion flows through its code each year, so the cost of missing an error is extremely high. Yet Minions run with full permissions and no confirmation prompts. The reason is that every Minion works in an isolated development environment, CI runs at most two rounds, and every PR gets human review. Stripe makes "discovering errors" extremely cheap, which is why it can afford to skip the "ask" step.

Codex is the opposite: it can pause a conversation for a round to wait for human approval. Neither approach is wrong; they simply answer the two questions differently.

The same lens explains other divergences: Cognition keeps a single writer, while Manus runs sub-agents in parallel, but neither allows parallel agents to write to shared state without a coordinator. The agents that operate directly on the system of record — Agentforce, Sidekick, Sierra, Fin, Harvey, Hebbia — are in a position where errors are hard to detect in real time and costly to miss, so they spend resources on deterministic gates, offline evaluation, rollback, and audit trails.

System of record refers to the core system in an organization that serves as the authoritative source of data, such as a CRM, ERP, order management system, or electronic medical record. Operations that write to such systems are persistent and high-impact: once incorrect data enters, it may trigger downstream processes, affect decisions across multiple parties, and be far more expensive to correct than fixing an error in a sandbox or draft environment. Therefore, agents that operate directly on systems of record must be equipped with deterministic gates, audit trails, and rollback mechanisms, rather than relying on the model's improvisation. This also explains why teams such as Agentforce, Sidekick, Sierra, Fin, Harvey, and Hebbia concentrate resources on offline evaluation and human review — once their errors are missed, the cost is irreversible.

An Unexpected Gap: No One Publishes "Cost per Successful Task"

The most surprising finding for the researcher was this: none of the 12 teams publishes the "cost per successful task." There are plenty of numbers on weekly PR counts and single-run accuracy. Fin publishes pass^k, Cursor publishes keep rate, Shopify publishes judge agreement. But almost no one states how much a task that actually worked (i.e., "succeeded") cost.

This leaves a key question: if you don't know the cost per successful task, you cannot judge whether the agent is truly more economical than a human. The researcher himself uses this metric to measure the AI research agent he is building, including the cost per accepted draft.

pass^k is a metric for evaluating agent reliability that measures the probability that the same task succeeds every time when run k times. Unlike single-run accuracy, pass^k emphasizes stability: an agent with a 90% single-run success rate may have only about a 59% chance of passing all runs when k=5, exposing the risk of occasional failure. For production environments, reliability matters more than peak capability, because users will repeatedly use the same feature. Fin publishes pass^k precisely to show its agent's ability to deliver consistent performance across repeated runs, rather than only showing results under the best conditions.

Limitations to Keep in Mind

The researcher openly acknowledges that 66 of the 83 claims he collected come from companies' own accounts, and that he did not actually run these harnesses himself. In addition, the first draft of this analysis came from an AI research agent he is building; he edited it and publicly measures that agent's cost.

For teams running agents, he leaves two questions: Do you track the cost per successful task, or a metric close to it? How do you define "success"? If you run one of these 12 and his description is wrong, what did you omit?

The full comparison, along with each claim and its date and source link, can be viewed on his website.

One-line Summary

Harness design consensus, sources of divergence, and the cost-evaluation gap across 12 production-grade agent teams.

Paragraph Summary

Based on a Reddit researcher's analysis of public material from 12 production agent teams, the article argues that the harness layer beyond the model is what determines agent performance. The 12 teams are highly consistent in three practices: progressively loading context on demand, putting deterministic steps into code, and verifying with tools other than the generator. Divergence between teams can almost always be explained by two questions: how cheap it is to discover an error, and how costly it is to miss one. Stripe allows Minions to run with full permissions because isolated environments and CI make error discovery extremely cheap, while Codex retains human approval. The article also points out a key gap: no team publishes the cost per successful task, making it impossible to judge whether agents are truly more economical than humans. The research is based on companies' self-reports, so conclusions should be treated with caution.