Same Model, 22-Point Score Gap: Agent Harness Is the Key Variable

One model, the same set of weights—yet simply swapping out the surrounding scaffolding can open up a 22-percentage-point gap on Terminal-Bench 2.0. This is a set of data that recently sparked discussion in the agent developer community: Opus 4.6's score swings from 60.45% to 82.02% across different harnesses, while GPT-5.5 ranges from 66.07% to 88.09%.

A developer who has long built agents in-house distilled their research into a practical guide, and the core point is direct: what most people underestimate isn't the model, but the entire set of things surrounding it—the loop, tool calls, context management, permissions, sandboxing, and tracing. This set of things now has a collective name: harness.

What Exactly Is a Harness

As defined in this guide, a harness is the execution layer that wraps around the model and drives the agent to complete tasks. It's not the same as a framework or a runtime: a framework provides abstractions and components, a runtime handles the execution environment, but a harness cares about "how to advance this turn, which tool to call next, how to trim context, and how to retry on failure."

The author lays out a five-layer architecture: model, harness, tools, environment, and observability. The model handles reasoning, the tools handle actions, the environment provides the sandbox and filesystem, and observability records the inputs, outputs, and cost of every step. The harness sits in the middle—it's the layer that strings these together into a closed loop.

Terminal-Bench 2.0 is a benchmark for evaluating AI agents on multi-step tasks in real terminal environments. It simulates the shell operations developers perform daily, requiring agents to achieve goals through command-line tools, file editing, process management, and other means rather than single-turn Q&A. Unlike benchmarks focused on code completion or static reasoning, Terminal-Bench 2.0 inherently emphasizes long-chain loop progression, tool-call accuracy, and context window management—which is precisely why harness quality gets significantly amplified. SWE-Bench Pro V2, meanwhile, focuses on software engineering repair tasks, with different evaluation granularity and statistical methodology, where model capability dominates. The difference in conclusions between the two benchmarks essentially reflects how task type drives sensitivity to the harness.

Same Model, Why Can Scores Differ by 22 Points

The official Terminal-Bench 2.0 leaderboard data illustrates the point best. Opus 4.6's score range is 60.45% to 82.02%, and GPT-5.5's is 66.07% to 88.09%—the gap between upper and lower bounds exceeds 20 points in both cases, while the model itself hasn't changed.

LangChain offers a similar case: Deep Agents, on GPT-5.2-Codex, improved its score from 52.8% to 66.5% simply by restructuring the harness.

These numbers point to one judgment: on complex task-oriented benchmarks, engineering implementation quality may be as important as model choice—or even more so.

But Don't Rush to Say "Harness Is Everything"

The author candidly presents counter-evidence in the guide. The variance analysis in the SWE-Bench Pro V2 paper shows that the scaffold explains only about 0.29% of the result variance, while the model explains 2.65%.

These two conclusions seem contradictory but each holds on its own: Terminal-Bench 2.0 emphasizes multi-step task completion in terminal environments, which is extremely sensitive to loop design, tool-call accuracy, and context window management; whereas SWE-Bench Pro V2 measures different task types with different statistical methodology, where model capability remains the dominant factor. Saying the harness is "everything" is an exaggeration, but saying it "doesn't matter" is equally a misjudgment. The more reasonable statement is: on long-chain, multi-tool, highly interactive tasks, the leverage effect of the harness is significantly amplified.

What the Guide Covers: From Concepts to Runnable Code

The guide's structure starts from foundational concepts and progresses layer by layer to intermediate hands-on material:

• A step-by-step breakdown of the agent loop, with flowcharts that explain each link of "observe—think—act—observe again"
• The five-layer architecture: model, harness, tools, environment, observability
• A side-by-side comparison of mainstream solutions: Claude Agent SDK, OpenAI Agents SDK, Google ADK, Deep Agents, MS Agent Framework, Pydantic AI, OpenHands, and the various CLIs
• A minimal runnable Python harness loop, explained with line-by-line comments
• Engineering details: cost tracking per run, permission models, sandbox strategies

The minimal Python harness loop deserves special mention. Many developers start by directly applying an existing framework, and as a result have little awareness of the token consumption inside the loop, the handling of tool return values, or the boundaries of failure retries. Writing a minimal loop by hand is the fastest way to make these hidden costs explicit.

What Signals Are Hidden in the Comparison List

The side-by-side comparison of eight solutions is itself a highly informative choice. Claude Agent SDK and OpenAI Agents SDK represent the native paths of two model vendors; Google ADK targets its own ecosystem; Deep Agents takes a composable middle-layer approach; MS Agent Framework and Pydantic AI focus on type safety and engineering discipline respectively; OpenHands and the various CLIs are closer to developers' everyday terminal workflows.

This list shows that the harness field hasn't converged on a single standard, and vendors differ noticeably in their trade-offs around abstraction level, permission models, and observability interfaces. When choosing, the real question isn't "which is most popular," but "how long is my task chain, how fine-grained does my tracing need to be, and how far does sandbox isolation need to go."

Claude Agent SDK is Anthropic's official agent-building toolkit, with built-in capabilities like tool calling, context management, and sub-agent orchestration; OpenAI Agents SDK is OpenAI's corresponding solution, emphasizing lightweight orchestration and the handoff mechanism; Google ADK (Agent Development Kit) targets the Gemini ecosystem, offering multi-agent collaboration and deployment support. Deep Agents comes from LangChain and is positioned as a composable middle layer, providing planning and subtask decomposition between the model and tools. MS Agent Framework and Pydantic AI approach from the angles of enterprise integration and type safety, respectively. The differences in abstraction level among these solutions directly determine how much control developers have over the internal details of the loop—the higher the level of encapsulation, the less effort it takes, but the harder it is to fine-tune.

Cost, Permissions, and Sandboxing: Three Underestimated Things

The guide places cost tracking, permission control, and sandbox isolation at the end of the hands-on section, and these three are exactly where things most easily go wrong between demo and production.

Cost tracking per run isn't just about money—it's also a diagnostic tool for spotting anomalies in the loop; a sudden spike in token consumption often means something has gone wrong with context management. The permission model determines which resources the agent can touch, and in terminal-type tasks this step is directly tied to security boundaries. The sandbox is the last gate that limits the blast radius when something goes wrong.

None of these three fall within the scope of model capability, yet they determine whether an agent can go from an experimental script to a deliverable product. Returning to that 22-point gap, the answer may lie right here: behind the score differences is the accumulation of these engineering details.

The full original guide is titled "The Harness Is Half the Agent."

One-line Summary

Same model, different harness, 22-point score gap—but the harness isn't everything.

Paragraph Summary

The article explores the harness (execution layer) that is often underestimated in agent development. Official Terminal-Bench 2.0 data shows that Opus 4.6 and GPT-5.5 both have score gaps exceeding 20 points across different harnesses, and LangChain also improved Deep Agents' score by 13.7 percentage points by restructuring the harness. But the author also presents counter-evidence: in the SWE-Bench Pro V2 paper, the scaffold explains only about 0.29% of the result variance. The conclusion is that the harness's leverage effect is amplified on long-chain, multi-tool tasks—but it doesn't determine everything. The article also covers the five-layer architecture, a comparison of eight mainstream solutions, a minimal Python loop, and three underestimated engineering details: cost tracking, permissions, and sandboxing.