Harness Engineering: The LLM Is the Brain, the Harness Is the Body

What We're Overlooking When We Talk About LLMs

A post on HN with only 39 points and 17 comments had a title so precise it made you stop: Harness Engineering 101: The LLM is the brain, the harness is the body. No big-company PR release, no funding news—but this is exactly the concept that most needs systematic discussion in the AI application layer right now.

A large model on its own doesn't solve any end-to-end problem. It won't remember the context from your previous turn (unless the framework remembers it for it), won't call APIs, won't read a database without tools, and certainly won't proactively retry failed operations. All of this engineering layer that "lets the model actually do work" is the harness—or, to put it more precisely, the "scaffolding" or "execution framework." The article uses a minimalist metaphor: the LLM is the brain, the harness is the body. No matter how smart the brain is, without a body to perceive and act, it's just a floating reasoning machine.

What Exactly Does the Harness Include

If we treat the LLM as a stateless function, the harness is the state-management and tool-execution system wrapped around it. Broken down, it includes at least four layers:

The first layer is context orchestration. Each model call can only see the current token window. Who decides what to put in, in what order, when to summarize and compress, and when to discard old information—all of that is the harness's job. The same model, under two different context-management strategies, can perform several tiers apart.

The second layer is the tool interface. Function calling, MCP, code interpreters, browser control—the model itself only outputs a structured call intent. Actually executing it, handling exceptions, and formatting the result back into a model-readable form is all done by the harness. How tool descriptions are written, whether return results should be truncated, and in what format error messages are passed back—these details directly determine whether the model can use tools correctly.

The third layer is control flow. How many steps a task needs, when to continue the loop, when to stop and ask a human, how many times to retry on failure, whether to run in parallel or serially—these are not part of model reasoning, but rather the harness's decision logic. Patterns like ReAct, Plan-and-Execute, and Reflexion are essentially architectural choices at the harness layer.

The fourth layer is state and memory. Conversation history, task progress, intermediate artifacts, and long-term memory across sessions. Models have no persistence capability; all "remembering" is the harness reading and writing to external storage.

Together, these four layers form a runtime system. The model can be swapped at any time—from GPT to Claude to open-source models—but the harness often needs to be rewritten or significantly adjusted.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic in late 2024 to standardize how LLM applications connect to external tools and data sources. The traditional approach is for each application to define its own function-calling format and implement its own tool adapters, leading to a fragmented tool ecosystem. MCP attempts to serve as the "USB interface" for this layer: tool providers expose capabilities according to the protocol, and the harness discovers and calls them according to the protocol, decoupling both sides. For harness engineering, the value of MCP lies in standardizing the tool interface layer, making the same tool reusable across different frameworks and reducing migration costs when switching models or frameworks. However, MCP mainly solves the problem of "how to connect." How tool descriptions are written, how return results are trimmed, and how errors are normalized remain engineering decisions belonging to the harness itself.

ReAct, Plan-and-Execute, and Reflexion are three common agent control-flow patterns, differing in "who decides the next step." ReAct has the model alternate between reasoning and action at each step, with the harness executing the action and feeding the observation back, looping until the model gives a termination signal—flexible, but the number of steps is unpredictable. Plan-and-Execute first has the model produce a complete plan in one go, and then the harness executes it step by step—highly controllable, but the whole thing fails when plan quality is poor. Reflexion introduces a self-reflection step after a failed execution, writing the reasons for failure as text feedback into the next attempt; in essence, it maintains an "experience memory" at the harness layer. None of the three are model capabilities; they are the harness's orchestration choices for loop structure, termination conditions, and feedback loops, and real products often mix them.

Why It Deserves to Be Treated as an Engineering Discipline

One point repeatedly echoed in the HN comments: most people spend 90% of their effort on choosing models and writing prompts, but what really determines product experience is the harness. This judgment deserves serious consideration.

The same model performs fine in a chat scenario, but put it into a task that requires calling 5 tools, handling 3 failure modes, and maintaining multi-turn state, and problems immediately surface. Where do the problems come from? Often it's not that the model "got dumber," but that the harness didn't give the model enough support: vague tool descriptions, raw error messages thrown straight through, brute-force truncation once context exceeds the window, and no termination conditions leading to infinite loops.

From an engineering perspective, harness design has its own set of trade-offs:

• Control granularity: Do you let the model freely decide each step, or does the harness orchestrate a fixed flow? The former is flexible but uncontrollable; the latter is stable but rigid.
• Error recovery: Do you throw exceptions back to the model as-is, or does the harness first do a layer of normalization?
• Observability: Is there a complete trace of the input and output of every call, and the latency and results of every tool call? A harness without traces is nearly impossible to debug.
• Cost and latency: Parallel calls, caching, and tiered model routing are all decisions made at the harness layer.

The answers to these questions aren't written in any model's system card, but they determine whether the end user experiences an "intelligent assistant" or an "artificial idiot."

Observability is even more critical in LLM applications than in traditional backends, because failures often aren't thrown exceptions but the model "quietly doing the wrong thing." A complete trace needs to record at least: the full input tokens and output of every model call, the parameters and returns of every tool call, the latency and token consumption of every step, and the control-flow jump path. Without these, debugging is just guessing—the same bug may not reproduce under different temperature parameters and context. In engineering practice, traces are usually organized in three levels—"session-turn-step"—and support replay by failed step. Observability is also a prerequisite for cost optimization: only by seeing the token distribution of each call can you judge which context to compress and which tool results to cache.

An Underrated Conclusion

The body metaphor in the original title boils down to one sentence: the upper bound of a model's intelligence is determined by the model, but the upper bound of product experience is determined by the harness. Current model capabilities are already sufficient to support a large number of practical tasks; the bottleneck often lies in the engineering quality of the execution framework.

For teams building AI applications, this means the direction of investment needs to be reallocated—rather than repeatedly comparing benchmark scores, it's better to spend time on context strategy, tool interface design, error handling, and observability. These things aren't sexy and don't easily make it into a launch announcement, but they are the threshold that takes a model from "demo-usable" to "production-usable."

The HN post didn't get much traction, but the framework itself deserves to be seen by more people doing AI engineering.

One-line Summary

The LLM is just the brain; what really determines the ceiling of AI product experience is harness engineering.

Paragraph Summary

Drawing on a discussion post from HN, the article introduces the concept of "Harness Engineering," pointing out that a large model itself is a stateless reasoning engine that won't remember context, call tools, or handle failures—what actually lets the model do work is an execution framework wrapped around it. The article breaks the harness into four layers: context orchestration, tool interfaces, control-flow decisions, and state and memory management, emphasizing that the same model can perform vastly differently under different harnesses. The author argues that most teams over-invest effort in choosing models and writing prompts, while the real bottleneck in product experience lies in engineering quality such as tool descriptions, error recovery, observability, and cost and latency, and calls on teams to reallocate their investment direction.