Demystifying Claude Code's Harness in 60 Lines of Python: The Core of AI Coding Tool Performance Differences

What Is a Harness? The Overlooked Core of AI Coding

In the world of AI coding, we've grown accustomed to a variety of vague terms—agentic coding, vibe coding—and now there's another concept to understand: the Harness. In his latest deep-dive video, Theo (t3.gg) makes it clear that the Harness is a very concrete technical concept that has a decisive impact on the quality of code you get from AI tools.

The rise of the Harness concept is directly tied to the wave of AI coding tools evolving from "code completion" toward "autonomous agents." In the early days after GitHub Copilot launched in 2021, bringing LLMs into the programming workflow, the role of the tooling layer was relatively simple—mainly passing editor context to the model and filling in completion suggestions. The Harness concept in its truest sense emerged around 2023 as the agentic coding paradigm matured, driven by a qualitative leap in model capability: once models like GPT-4 and Claude 2 demonstrated reliable instruction-following and structured output, letting a model "autonomously operate the file system" moved from theoretically feasible to engineering practice.

It's worth noting that a Harness occupies a position in the technical architecture analogous to the Runtime Environment in traditional software engineering. It's responsible not only for registering and dispatching tools, but also carries core duties like context management, security sandboxing (preventing the model from performing destructive operations), and error recovery (retry and fallback strategies when tool calls fail). From a layered architecture perspective, the Harness sits between the model API layer and the user interface layer, serving as the critical middleware that determines the reliability and predictability of the entire system—which is exactly why the same underlying model can exhibit such wildly different real-world performance across different Harnesses.

The most convincing data comes from Matt Mayer's independent benchmark: the same model performs dramatically differently across Harnesses. Take Opus, for example—it scores 77% in Claude Code, but jumps to 93% in Cursor. The only variable is the Harness. This figure alone shows that the Harness is far from a dispensable shell; it's the key factor determining the ceiling of a model's capabilities.

What exactly is a Harness? In the simplest terms: a Harness is the collection of tools and runtime environment an Agent depends on to operate. It's the underlying infrastructure layer that lets an AI actually "do things." Interestingly, Cursor, Claude Code, Codex CLI, and OpenCode are all Harnesses, whereas T3 Code and the Codex App are not—a distinction we'll explore in detail later.

LLMs Only Generate Text—How Tool Calling Changes Everything

To understand the value of a Harness, you first have to grasp one essential truth: all LLMs are fundamentally advanced autocomplete. You input text, and they predict the most likely next chunk of characters, over and over. The model itself cannot edit files, operate databases, connect to external services, or search the web—the only thing it can do is generate text.

So how do these models edit files and execute commands locally? The answer is Tool Calling. Tool calling is the core mechanism for extending modern LLM capabilities. It was first formally introduced by OpenAI in 2023 in the form of "Function Calling," after which major vendors like Anthropic and Google followed suit and established a de facto standard.

It's worth noting that from OpenAI's Function Calling interface, to Anthropic's Tool Use API, to Google's Function Declarations, the interface designs of these three major platforms have evolved over the past two years from going their separate ways to gradually converging—JSON Schema became the universal format for describing tool parameters. This standardization process dramatically lowered the cost of developing cross-platform Harnesses, and it's the underlying reason why a single Harness can relatively easily connect to multiple model providers at once. Behind this standardization lies a lesser-known driving force: the emergence of the Model Context Protocol (MCP)—this open protocol, proposed by Anthropic in late 2024, attempts to establish a higher-level interoperability standard at the tool-calling layer, bringing the complete lifecycle of "tool definition, invocation, and response" under a unified specification, so that different Harnesses can share the same tool ecosystem without having to reimplement adaptation layers for each model provider. The underlying principle is that during model training, large amounts of conversational data containing structured tool calls are introduced, teaching the model to recognize "when it should use a tool" and "how to format a tool-call request." At the implementation level, a tool call is essentially a special token sequence in the model's output, which the Harness layer is responsible for intercepting, parsing, and routing for execution.

The mechanism is quite clever: the model is told in advance in the system prompt which tools are available, for example a bash-call tool. When the model needs to use it, it wraps the command in specific syntax, such as <bash_call>ls -a</bash_call>, and then stops responding.

Something critical happens here: once the model outputs the tool-call syntax, it stops responding. At this point the conversation between the model and you is completely severed; the conversation history exists only on your local machine or server, and the model is in a "paused" state.

Next comes the Harness's moment to shine. The Harness receives the tool call and executes it with the most basic code—depending on your settings, it either runs it directly or requests authorization first (for example, prompting for confirmation before destructive operations like formatting a file). Once execution is complete, the Harness appends the tool output to the end of the conversation history, then sends a new request to the same model to let it continue working.

This loop of "model generates tool call → Harness executes → result is filled back into history → model is requested again" is known in the AI engineering field as the Agentic Loop or the ReAct Loop (Reasoning + Acting, alternating reasoning and action). The ReAct framework was formally proposed by Princeton University in 2022, formalizing the paradigm of alternating reasoning steps (Thought) with action steps (Action), and became the theoretical foundation for nearly all subsequent Agent frameworks.

The core insight of this framework is that pure "Chain-of-Thought" reasoning can only spin within the model's internal world, whereas only by interweaving reasoning with real-world actions (tool calls, environment interactions) can a model accomplish truly complex tasks. Mainstream Agent frameworks like LangChain, LlamaIndex, and AutoGen are all engineering implementations of this paradigm; they build on ReAct by further introducing mechanisms like memory management, multi-agent coordination, and tool routing optimization, but their most fundamental loop structure is essentially identical to the Harness you hand-wrote in 60 lines of Python.

Understanding this loop also carries an important engineering implication: each tool call is a complete API request, meaning token consumption, latency, and cost all grow linearly with the number of tool calls. A well-designed Harness makes trade-offs at the granularity of tool calls—too fine a granularity causes too many API round-trips, while too coarse a granularity increases the risk of a single call failing. This is precisely why commercial tools like Cursor specifically optimize "Parallel Tool Calling" capabilities, allowing the model to declare multiple parallel tool calls in a single response, which the Harness executes concurrently and fills back into the result together—significantly reducing latency without sacrificing accuracy. The "brain" responsible for all the work is paused and restarted with every tool call, and this is exactly how nearly all current AI coding tools fundamentally operate.

Context Is King: Why Large Contexts Actually Make Models Dumber

Theo particularly emphasizes a counterintuitive point: if information isn't in the conversation history, the model doesn't know it. This doesn't apply to general knowledge like TypeScript syntax, but for your private codebase, the model knows nothing—unless it actively explores through tool calls, or you "inject" context to the top of the history in advance via files like CLAUDE.md or AGENTS.md.

When you open Claude Code in a project directory, it knows nothing about the codebase. It calls search tools to find matching files, reads package.json as a starting point, then reads related source files, dumping all of that output into the context. And if you've provided a CLAUDE.md in advance, the model might skip the tool calls entirely.

Here Theo debunks a once-popular theory: the belief that if a codebase was too large to fit into the context window, it couldn't be worked with. This gave rise to tools like RepoMix, which compress an entire codebase into a single XML file to feed the model. But this creates the worst possible "needle in a haystack" problem.

The phenomenon of large context windows causing performance degradation has been systematically studied in academia. Stanford University's 2023 paper "Lost in the Middle" was the first to quantitatively describe this pattern: when key information is located in the middle of a long context, the model's retrieval accuracy is significantly lower than when the information is at the beginning or end—structurally quite similar to the "primacy effect" and "recency effect" in human memory. The deeper cause of this finding points to the attention mechanism of the Transformer architecture: in extremely long sequences, tokens in the middle positions often receive relatively lower attention weights in multi-head attention computation. This is an inherent bias at the architectural level, not something that can be fully overcome by simply adjusting the prompt.

Interestingly, this problem is not unsolvable. Recent research directions include: positional encoding optimization (improvements like RoPE and ALiBi attempt to make the model maintain more uniform attention allocation across different positions); context compression techniques (such as Anthropic's Prompt Caching mechanism, which reduces redundant computation cost by caching recurring long prefixes, combined with summary compression to streamline conversation history); and sparse attention mechanisms (letting the model compute full attention only over "important" tokens while using approximate computation for the rest). These technical paths all attempt to mitigate the "Lost in the Middle" effect at the architectural level, but as of now, no single approach can completely eliminate this bias in all scenarios—which is why precise context management remains one of the most critical engineering challenges in Harness design.

It's precisely based on this understanding that Retrieval Augmented Generation (RAG) technology rapidly rose as a systematic solution. RAG's core idea is to split the text in a knowledge base into small chunks, convert each chunk into a high-dimensional vector using an Embedding Model, and store them in a vector database (such as Pinecone, Weaviate, Chroma, etc.). When a user asks a question, the system first vectorizes the question, then performs approximate nearest neighbor (ANN) retrieval in the database, taking only the few fragments with the highest semantic relevance to inject into the context—rather than brute-force stuffing the entire knowledge base. This mechanism fundamentally sidesteps the "needle in a haystack" dilemma: what enters the model's context is always highly concentrated relevant information, not redundant full data. For private codebases, a modern Harness's RAG implementation must additionally handle the structured semantics of code—structural information like function signatures, class hierarchies, and module dependencies can express semantic relationships between code better than plain text, which is precisely the core difference between code-specific embedding models (like Voyage Code, CodeBERT) and general-purpose text embedding models.

Theo uses a brilliant analogy: imagine your memory is reset every 30 seconds. You're asked to fix a bug, and knowing your brain is about to reset, you initiate a search—the moment the search begins, your brain is wiped, and after the search completes, your brain restarts, retaining only the history record… and so on repeatedly. Stuffing an entire codebase into a brain like this that keeps resetting is both expensive and inaccurate.

The data confirms this too: when Sonnet's context breaks past the 50k to 100k token range, its accuracy in finding a repeated word within the context window plummets by nearly 50%. Large contexts make models dumber. Therefore, the real value of a modern Harness lies in providing the model with tools to precisely build the context it needs on its own, rather than cramming everything in all at once.

60 Lines of Python: Build a Harness with Your Own Hands

Theo cites Mihail's article "The Emperor Has No Clothes," whose central point is: tools like Claude Code aren't complex—their core is roughly only about 200 lines of straightforward Python. The entire sequence of events is: you send a message → the LLM decides to use a tool and returns a structured call → your program (the Harness) executes it locally → the result is passed back to the LLM → the LLM continues based on the new context.

The core needs only three tools: read file (letting the LLM see the code), list files (navigating the project structure), and edit file (actually making changes). A production-grade Agent will also have grep, bash, web search, and so on, but the most basic version doesn't need them.

The code structure is quite concise: load environment variables, initialize the Anthropic SDK client, resolve absolute paths (making it easier for the model to write valid commands), and implement the three tool functions. read_file returns the file contents as JSON; list_files traverses the directory and returns file names and types; edit_file uses old_string and new_string to make replacements, and if old_string is empty it creates a new file.

The key is letting the model know these tools exist. Since Python doesn't have type signatures like TypeScript, you need to describe each tool's functionality and parameters in detail through comments. These descriptions, along with the tool list, are assembled into the system prompt—essentially telling the model: "These are the tools you can call; when needed, reply with a line of tool:toolName plus JSON parameters." This system prompt constitutes the vast majority of a Harness's core logic.

The system prompt is the core interface between the Harness and the model, and its design quality directly determines the predictability and stability of model behavior. Professional System Prompt Engineering has gradually evolved into an independent engineering discipline, spanning multiple dimensions such as role definition, constraints, tool description formats, output specifications, and edge case handling. The core difficulty of this discipline lies not in "writing beautiful descriptions" but in cross-model robustness: the behavior of the same system prompt on GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro can differ by more than 30%, because different models were exposed to different distributions of instruction data during pretraining and RLHF, giving them entirely different "interpretation tendencies" for the same semantic natural-language description.

At the practical level, the industry has developed several validated structured paradigms for system prompts. Anthropic internally calls its structured approach to system prompt design "Constitutional AI Prompting," emphasizing constraining model behavior by explicitly enumerating Principles rather than relying on implicit natural-language tone. OpenAI, in its GPT-4o technical report, disclosed a layered structure for its system prompts—nesting Global Constraints, Persona Definition, Tool Specifications, and Output Format in layers rather than laying them out as one continuous block of text. The advantage of layered nesting is that when the model parses it, it assigns higher attention weights to the earlier layers, which aligns with the positional bias of the attention mechanism—placing the most important constraints first can statistically reduce the probability of the model "forgetting" key rules.

An often-overlooked detail of system prompt design is the wording of Negative Constraints. Research shows that "don't do X" and "always do Y instead of X" have significantly different effects on model behavior: the former requires the model to continuously suppress a certain output pattern during generation, while the latter indirectly avoids incorrect behavior by guiding the model toward the correct path. In the context of tool descriptions, this means that instead of writing "don't output file contents directly, call the read_file tool," it's better to write "when you need to view a file, call the read_file tool and pass in the path parameter"—the latter phrasing better aligns with the "execute step-by-step" behavior pattern the model learned during RLHF. This is precisely why commercial Harnesses need to maintain and iterate separate prompt versions for each supported model; this work seems tedious, but it's the key to differences in product quality.

The rest is parsing and looping: after the model stops responding, find the lines starting with tool:, parse out the tool name and parameters, retrieve the corresponding function from the registry and execute it, append the result to the history as a message, then re-invoke the model in the loop. In his demo, Theo even found that keeping only the bash tool was sufficient—the model would repeatedly call bash to accomplish all tasks, trimming the code down to 75 lines, and even fewer after removing the environment variable handling.

This reveals a stunning fact: giving an AI the ability to operate a computer only requires giving it a tool that can pass bash commands. These models have seen large amounts of "fake conversation history" containing tool calls during training, and they inherently know how to handle tool-call responses.

Why Is Cursor Stronger? The Secret of System Prompts and Two Ultimate Questions

If the Harness is so simple, why can Cursor make the same model perform so much better? Theo's answer is: because they test more. The shape of the tools, the wording of the system prompt, the details of tool descriptions—all greatly affect the final result. Cursor invests enormous effort in customizing its Harness, even having dedicated people repeatedly fine-tune system prompts and tool descriptions when a new model is released or when they get early access, until the model works roughly as expected.

Even more interesting, Theo demonstrated the phenomenon of "lying to the model." He changed the read_file tool's description to "deprecated, please use the bash tool instead," and Sonnet obediently switched to using bash; he even made the read tool return "print hello world" no matter what file was read, and the model believingly reported that the code was only one line. The model doesn't know what the tool actually does—it only knows what you tell it. You can call a tool bash while it does something else, and you can fabricate responses to make two models converse with each other without them realizing it.

This phenomenon reveals a core issue in the AI security field: the trust boundary problem of tool calls. The model's unconditional trust in tool responses means that a maliciously constructed Harness can systematically deceive the model—known in security research as a "Tool Poisoning Attack." Several security research papers in 2024 documented this attack vector: by controlling the content of tool responses, an attacker can induce the model to perform operations it otherwise wouldn't, or leak sensitive information from the context. This is also why the transparency of open-source Harness frameworks (like the open-source version of Claude Code and OpenCode) is increasingly valued in enterprise procurement decisions—users can audit every line of the Harness's code to confirm that tool behavior fully matches its description, rather than relying on blind trust in a commercial product.

Different models also react very differently to the same descriptions. Given the same "deprecated" label, Gemini 2.5 Pro aggressively "uses bash for everything," while Claude is more cautious. Behind this difference lie the different "personality tendencies" each model formed during the RLHF (Reinforcement Learning from Human Feedback) training stage.

The complete RLHF process comprises three key stages: first is supervised fine-tuning (SFT), initializing the pretrained model with high-quality human demonstration data; second is reward model training (Reward Modeling), training a preference predictor by having annotators make pairwise comparisons of multiple model outputs; and finally policy optimization (Policy Optimization), usually using the PPO (Proximal Policy Optimization) algorithm to continuously adjust the main model's output distribution using the reward model's scoring signal. In Google's large-scale internal deployment scenarios, the Gemini series' annotator pool and scoring standards were calibrated to lean more toward "efficient execution" instruction-following; whereas Anthropic's Constitutional AI framework adds an AI self-critique loop on top of RLHF, letting the model actively check and revise its answers against a set of principles after generating them—a mechanism that behaviorally manifests as a conservative tendency when facing ambiguous instructions.

It's worth noting that RLHF isn't just about making models "more obedient"—it's essentially about continuously and directionally adjusting the model's output distribution through human preference data. The choice of annotator pool, scoring standards, and reward model design by different companies all leave measurable imprints on the model's final "personality." The recent Direct Preference Optimization (DPO) technique attempts to bypass a separate reward model, directly fine-tuning the pretrained model with preference-pair data—an approach with significant advantages in computational efficiency, though whether its fineness in "personality shaping" can match the complete RLHF process is still debated in academia. This means Cursor must rewrite tool descriptions separately for each supported model—whereas Anthropic may not have changed much since this code was first written, which is precisely where the gap lies.

As for what T3 Code is? Theo admits it's not a Harness, because it provides no tools—no bash tool, no read tool. When you select a Claude model in T3 Code, it actually invokes the Claude Code Harness already installed on your machine; selecting Codex relies on Codex CLI. T3 Code is just a beautiful UI layer on top of a Harness. This also responds to the criticism that it's "just a wrapper"—building a Harness itself isn't hard; what's hard is crafting an excellent product experience around it.

Key Takeaways