Harness Engineering: The Secret to Tripling the Same Model's Score

The same frontier model, in the same puzzle game, saw its score jump from 13.3% to 38.3% purely because of differences in the surrounding code and settings. This gap isn't due to the model's capabilities—it's the "harness," the layer of engineering code wrapped around the model. If you're building AI agents, understanding and optimizing the harness is your core job.

What Is a Harness?

A harness literally means "tack" (horse gear). No matter how powerful a horse is, without a harness the rider could be thrown off and the horse would run wild; with a harness, the rider can control that power. The model itself only replies or requests tool calls, and the harness is the code around it: executing tools, carrying the conversation, deciding when work stops.

OpenAI gave a clear example in its report on ARK AGI 3 (a puzzle benchmark from ARC): it turned on two settings in its own harness—preserving the model's reasoning across steps, and summarizing history rather than truncating it directly. As a result, with the same number of output tokens, a better harness let the model take more actions and clear more levels.

The core takeaway from this case: when your agent underperforms, check the surrounding code first, rather than rushing to rewrite prompts.

ARC (Abstraction and Reasoning Corpus) is a general intelligence benchmark proposed by François Chollet in 2019, consisting of a series of visual grid puzzles. Each puzzle provides several input-output examples and requires inducing the transformation rule from the examples and applying it to new inputs. Its design deliberately targets "few-shot induction," a human strength, posing a challenge to models that rely on large-scale pretraining; it is therefore often used to measure a model's abstract reasoning and generalization ability rather than memory or language fluency. ARK AGI 3 is one of the evaluation attempts built around this benchmark, and its report is frequently cited because it isolates the variable of "same model, different surrounding engineering" relatively cleanly, making score differences more directly attributable to the harness rather than model weights.

The Seven Components of a Harness

The video author summarizes seven parts with the mnemonic "HARNESS," each letter corresponding to a component:

• Heartbeat (loop): Alternating between calling the model and executing the model's requests, until a final answer is obtained or a limit is hit.
• Actions: The agent's "hands," i.e., the tool calls that let it act in the real world.
• Reads (context): What the model can see at each step, determined by the harness.
• Notes (memory): Storing progress outside the model's view, so work survives across steps and sessions, and putting it back into context upon retrieval.
• Enforce (permissions and guardrails): Three things—who can do what, what running code can touch, and what is checked on the way in and out.
• Sanity check (verification): A check that compares the answer against the task before the agent hands back its answer.
• Sight (observability): Recording steps so that when something goes wrong you can see where it went wrong.

These seven parts form the complete framework of harness engineering. The author uses a refund support agent as an example to demonstrate how each component is implemented.

The Refund Agent Example: The Seven Components in Practice

Loop and Actions

A loop can keep running without making progress, so you need a step limit, and a response when the same failed action repeats. The author added a loop that calls the model with the conversation each time and sets a step cap; when the same call fails twice in a row, a hint is inserted asking for a different approach, but still within the original step limit.

A tool call is a named structured request, and the code must check and execute it. If execution fails without useful feedback, the next call lacks clues to correct the request. So you need to add input checks, run the tool inside a try block, capture the error as a readable result, then append it to the conversation and link it back to the original call—only then can a real model attempt a correction based on it.

Context and Memory

As the run progresses, history keeps growing and is discarded when necessary, and the next reply may lose information that prior work depended on. OpenAI's second setting—preserving the model's reasoning across steps—is essentially also storing information. The author's approach: make the conversation list explicit, add a line to clear old tool results that are no longer needed; other techniques include on-demand loading and sub-agent summarization. Long-running agents put more pressure on context; this is a separate topic.

Without somewhere to store progress, subsequent calls lose the information needed to continue. The author retrieves saved notes back into context at the start, and saves progress notes before exiting. For the refund agent, notes can retain the order being processed and the results confirmed by tools.

Compaction (history compression/summarization) refers to the practice, when the conversation history approaches the context window limit, of not simply discarding the earliest messages but instead having the model summarize the existing history into a shorter summary that preserves key facts and decisions, then using that summary as a new starting point to continue running. Its essential difference from "truncation" is that truncation loses information indiscriminately—the model may forget the task goal, confirmed parameters, or paths that failed; summarization tries to preserve the parts useful for subsequent decisions. The cost is that summarization itself requires additional model calls and may introduce information loss or bias. OpenAI's experiments in ARK AGI 3 showed that, under the same output token budget, using summarization rather than truncation lets the model complete more effective actions, indicating that context management strategy and model capability are multiplicative rather than additive.

Permissions, Sandboxing, and Guardrails

Without these controls, actions could exceed user permissions, running code could touch resources it shouldn't, or disallowed content could get through. The author adds a permission check before dispatch, deciding whether that user can request this action on this order; execution is routed through a sandbox, and a teaching stand-in checks file paths and hosts; plus a guardrail that checks the answer against application rules. Guardrails can also check incoming requests and tool calls. The sandbox does not decide whether the customer is entitled to a refund—that is the permission layer's job.

A sandbox is an isolated execution environment that lets agent-generated code or tool calls access only explicitly permitted resources, such as restricting readable/writable file paths, limiting accessible hosts and ports, and constraining available system calls. Its goal is to limit the "blast radius": even if the model is misled by prompt injection or produces erroneous code, the impact is confined to the isolation zone. It should be emphasized that the sandbox solves the question of "what can be touched," not "should it be done"—the latter is the responsibility of the permission layer and guardrail layer. The three are often conflated, but the boundaries are clear: permissions determine whether a subject can initiate an action, the sandbox determines what that action can technically touch, and guardrails perform content-level interception at the input, tool-call, and output stages according to business rules.

Verification and Observability

The agent may stop early with work unfinished. In the refund example, it might miss part of the customer's request, or claim a refund that the tool never confirmed. So before exiting, check the available tool results against the request. If it fails, append feedback about what failed; if steps remain, send the answer back into the loop to retry, still within the original step limit. If the limit is exhausted, return an incomplete result.

Readable traces help you find where the run went wrong, rather than guessing from the final answer. The author adds trace lines around each step, recording the model's reply, tool requests, results, and checks under the same run identifier. The OpenAI team found the fix precisely by reading the puzzle attempts.

Don't Write from Scratch: The SDK vs. Build-Your-Own Trade-off

You don't need to write a harness from zero. The Claude Agent SDK and OpenAI Agents SDK both provide loops you can build on. The Claude Agent SDK includes context management, tools, permissions, and hooks, which let applications respond to events; the OpenAI Agents SDK likewise runs the loop and provides instructions, tools, guardrails, and lifecycle hooks. You still shape the harness through rules, tools, permissions, and hooks—these application decisions are always yours.

The author's advice: write your own when you need full control; building the loop yourself is also the best way to understand what happens between calls.

"Hooks" in an Agent SDK refer to the mechanism for registering callback functions at fixed lifecycle points in the agent loop, such as before a tool call, after a tool returns, after a model reply is generated, and at session end. Hooks let application code inject cross-cutting logic without modifying the SDK's internal loop: recording traces, performing permission checks, rewriting tool results, triggering external notifications, or forcing termination. They complement "guardrails"—guardrails are the decision rules, and hooks are where the rules are mounted and executed. Understanding when hooks fire is key to deciding whether a piece of harness logic should be written inside the loop or attached to a hook; placing it wrong causes checks to happen at the wrong stage, for example making a permission decision after the tool has already executed.

How to Improve Your Harness

The author's improvement path is pragmatic:

1. Run real tasks and read the run records, paying special attention to the steps before something went wrong.
2. Find the harness choice that likely explains the behavior: did results disappear? Did a tool fail, or is the loop repeating?
3. Change only one thing at a time, then re-measure on the same tasks using metrics that matter to the application.
4. Keep the changes that work, using the recorded steps to understand the difference.

As model capabilities improve, you also need to re-test the harness: check how inherited rules and helper code perform on your tasks, remove what the model no longer needs, add what the new model needs, and keep changes only after verifying the effect. The goal is "the simplest harness that works," letting every piece of code prove its value in run records you have read.

Key Takeaways

With the same model, the surrounding code determines the performance gap. When an agent gets stuck, read the run records and check what your harness carried into the next step. This gets closer to the root of the problem than repeatedly polishing prompts.

One-line Summary

With the same model, only the surrounding harness differs, and the score rises from 13.3% to 38.3%—the harness is the key to agent performance.

Paragraph Summary

The article points out that the same frontier model, in the same puzzle task, improved its score from 13.3% to 38.3% solely because of different surrounding engineering code (harness) settings, showing that the key variable in agent performance is often not the model itself. The author uses the "HARNESS" mnemonic to break down seven components: heartbeat loop, actions, reading context, notes/memory, permissions and guardrails, verification, and observability, and demonstrates each in practice with a refund support agent. The article also introduces two key OpenAI settings on ARK AGI 3—preserving reasoning across steps and history summarization—and offers an improvement path: read run records, change only one variable at a time, re-measure on the same tasks, and pursue the simplest harness that works.