Harness Engineering in Practice: Mastering the Four Core Principles and Implementation Methods

Why Harness Engineering Is the Core of Agent Development

Exceptional performance on complex problems, yet frequent failures on simple ones; unstable execution where minor issues snowball into major incidents — this is the biggest obstacle preventing AI Agents from becoming truly productive. The industry-recognized solution is Harness Engineering.

A consensus has emerged within the industry: what determines whether an AI agent is actually useful often isn't the model itself, but the constraints applied during Agent runtime. This concept is captured in a well-known formula:

Agent = Model + Harness

The model is the engine; Harness Engineering is the chassis, brakes, and steering wheel. Without the latter, even the most powerful engine will send you flying off the track faster. This formula has been repeatedly referenced in Anthropic's engineering blog and OpenAI's internal practices. Its core insight stems from a counterintuitive finding: in real production environments, the marginal contribution of model capability improvements to final output quality is diminishing, while the contribution of the engineering constraint system (i.e., the Harness) around the model to output stability is rising sharply. This aligns with the philosophy of "defensive programming" in traditional software engineering — you can't assume every component of the system will work perfectly; instead, you must architect failure paths and error-correction mechanisms at the system level.

This article draws on the hands-on development experience of the Bilibili creator "Jiutian" (Nine Days) team over the past six months, systematically outlining the four foundational theories of Harness Engineering and their practical implementation methods, along with two proprietary Harness systems — the Skill Distillation System and the Skill Optimization System.

The Four Foundational Theories of Harness Engineering

Synthesizing practices from OpenAI, Anthropic, and the broader community, the core of Harness Engineering encompasses four pillars: Documentation as Source of Truth, Mechanized Structural Constraints, Feedback Loops, and Entropy Management. Executing well on these four fronts will noticeably improve Agent performance.

Documentation as Source of Truth

Unlike humans, an AI agent has no long-term memory of a project — every new session starts as a blank slate. Documentation as Source of Truth addresses this problem by writing all relevant project information (tech stack, commands, conventions, forbidden zones, etc.) into a document that the agent automatically reads every time it starts up.

• In the Anthropic ecosystem, this is called CLAUDE.md
• In the OpenAI ecosystem, it's called AGENT.md

The codebase, together with this document, becomes the Agent's sole source of truth, replacing ad-hoc verbal instructions during each conversation.

The most common pitfall: trying to include everything. Writing the document into a hundreds-of-lines encyclopedia is a mistake. The context window is the scarcest resource — the more bloated the document, the more easily critical information gets buried. The Context Window is the maximum number of tokens a large language model can process in a single inference. Although the latest models like Claude 3.5 and GPT-4o support 128K or even 200K token contexts, this doesn't mean you can pile on information indefinitely. Research shows that large models suffer from a "Lost in the Middle" phenomenon — when the context is too long, the model's attention to information in the middle positions drops significantly, maintaining high recall only for content at the beginning and end. Therefore, document design must follow the principle of maximizing information density, placing the most critical constraints in the most prominent positions. The right approach is to treat it like a map, not an encyclopedia — keep the main file to about a hundred lines with a table of contents and the most critical constraints, while pushing detailed rules into subdirectories that the agent can navigate to on demand.

Mechanized Structural Constraints

Documentation alone isn't enough, because documents contain "suggestions," and agents — like humans — tend to ignore suggestions. This pillar upgrades critical constraints from suggestions to rules, hardcoding them into the execution flow for mandatory enforcement.

Structural constraints typically fall into two categories:

Mechanical Constraints: Deterministic interception using hooks or scripts. The hook mechanism borrows from concepts in Git Hooks and CI/CD pipelines. In Git, a pre-commit hook can automatically run lint checks or tests before code is committed; in Agent systems, hooks serve a similar but broader purpose — they can insert deterministic checking logic at multiple points: before tool calls (pre-action hook), after output generation (post-generation hook), during task transitions (transition hook), and more. The core advantage of hooks is that they are synchronous and blocking: once a violation is detected, the process is hard-interrupted, rather than merely "suggesting" the model make corrections as a prompt would. For example, automatically halting when the agent is about to execute a dangerous command, or running quality checks before code submission and hard-rejecting failures. Best suited for intercepting known risks that can be clearly described with rules.

Prompt Constraints: When risks are hidden in textual semantics and rule-based detection can't catch them, you need to invoke one large model to review another large model's output, relying on semantic understanding to identify issues like "this code effectively drops the database" or "this log line leaks a secret key" — problems that string matching can't catch.

The two work together to form a dual defense — mechanical constraints are the low-latency, high-speed sieve, while prompt constraints are the second safety net. Practice has repeatedly proven: constraining the agent within a reasonable solution space actually makes its output more stable and accurate.

Feedback Loop

Without feedback, an agent is like an archer shooting blindfolded — it doesn't know when it's wrong until everything is delivered and problems are exposed. The Feedback Loop ensures the Agent gets a "right or wrong" signal at every step.

Mature feedback is typically multi-layered:

• Immediate checks before and after tool calls
• Automated testing at submission time
• Runtime monitoring after deployment
• Independent review after task completion

The most critical principle: separate evaluation from generation. Large models have an incurable habit — when asked to grade themselves, they're always overly lenient. This phenomenon is known in academia as "Self-Consistency Bias" or "Self-Serving Bias." Multiple studies show that when the same large model both generates content and evaluates quality, it tends to give its own output high scores because generation and evaluation share the same internal representations and preference patterns. This is similar to the "endowment effect" in human psychology — people overvalue things they own. The reliable approach is to hand evaluation authority to a separate, independent agent dedicated to "finding flaws," or even use models from different providers for cross-evaluation to maximize the elimination of systematic bias. Tuning the skepticism level of an independent reviewer is far easier than keeping the producer perpetually self-aware.

Entropy Management

This is the most easily overlooked yet most original component. AI collaboration introduces a new kind of trouble: code and documentation don't suddenly crash — they slowly rot. Comments gradually drift from implementation, shortcuts bypassing conventions accumulate, different sessions leave behind conflicting styles, and historically accumulated dead features keep growing.

The concept of "entropy" is borrowed from the Second Law of Thermodynamics — in a closed system, entropy (i.e., disorder) always tends to increase. The software engineering field introduced the concepts of "Software Rot" and "Technical Debt" back in the 1990s, describing how codebases gradually degrade through continuous modification. In AI collaboration scenarios, this problem is dramatically amplified: human developers typically maintain a unified code style and mental model, while an AI Agent treats each session independently. Without a consistent "aesthetic memory" across sessions, style drift occurs far faster than in human collaboration. Martin Fowler's concept of "refactoring" is reinterpreted here as an automated, continuous entropy-cleaning process.

Entropy Management proactively combats this rot: codifying human taste into automated rules, continuously scanning for drift and cleaning it up immediately, and regularly compressing and archiving redundant information. Its essence lies in the word "continuous" — not waiting until things are unmanageable for a major overhaul, but running routinely like garbage collection.

Practical Case Study: Aligning AI Content with Aesthetic Preferences

Anyone who has used AI to batch-produce content has encountered this problem: the Agent writes quickly and prolifically, but something always feels off. You ask for clean and restrained, it gives you flashy flattery. You ask for honest illustrations, it fabricates a screenshot of software that doesn't exist. You ask it to explain jargon before using it, it opens with insider terminology. You ask for coherent prose paragraphs, it dumps a pile of bullet points.

Each of these is a minor issue on its own, but they're precisely what keeps AI-produced content stuck at "acceptable" without ever reaching "excellent." This isn't a capability problem — it's a taste problem. No matter how smart the model is, it can't guess personal preferences; no matter how elegant the architecture, it can't accommodate hundreds of trivial stylistic preferences.

How the Four Principles Work Together

Step 1: Establish the Source of Truth. The system maintains a dedicated experience library Global_Learnings.md to store aesthetic preferences and pitfall-avoidance rules. This document is intentionally kept separate from Claude's built-in memory — execution memory handles "how to get the job done," while aesthetic preferences handle "how to get the job done beautifully." Mixing the two types of information causes mutual interference.

Step 2: Use hooks for automatic logging. Lessons aren't conjured from thin air — they're captured on the spot by hooks during real execution scenarios: an illustration shows a fabricated page → log an entry; body text piles up bullet points instead of forming paragraphs → log an entry; Chinese text uses the wrong separator character → log an entry. The mechanism ensures that what should be recorded always gets recorded, rather than hoping the AI happens to jot something down on a good day.

Step 3: The feedback process makes lessons effective. Before starting work, the system predicts the current task type and proactively injects relevant past lessons; after completion, it tallies how many times each lesson was violated — which ones are still being repeatedly broken and which have been fully resolved, all crystal clear. Preference alignment transforms from "hoping for the best" into a measurable, trackable closed loop.

Step 4: Entropy Management keeps things sharp. What happens when too many lessons accumulate? Those no longer violated → cleaned out. Those only occasionally triggered → compressed into one or two sentences. Persistent hard cases → kept as-is or even strengthened in tone. The experience library thus stays "lean and sharp" rather than "bloated and sluggish."

Two Proprietary Harness Systems

In their public course, the team also built two core systems from scratch:

Skill Distillation System: Automatically captures high-frequency tasks during multi-turn conversations and distills them into reusable Skills. It fully automatically summarizes based on historical planning information with no manual intervention required. The core idea of the Skill Distillation System has deep connections to "Program Synthesis" and "Experience Replay." In reinforcement learning, experience replay stores an agent's historical interactions for later sampling and learning; Skill Distillation does something similar at a higher level of abstraction — extracting reusable task patterns from multi-turn conversation history and solidifying them into standardized execution templates.

Skill Auto-Optimization System: Capable of autonomously creating datasets from historical tasks during actual use, and independently testing and iteratively optimizing Skills. This system adopts an approach similar to "Automatic Prompt Engineering," building test datasets, running automated A/B comparisons, and iteratively improving based on evaluation metrics to form a continuous Skill evolution loop without human intervention. This is also a must-have core capability for industrial-grade Harness systems, and the key ability behind the recently trending Harness Agents — using Agents to optimize the Agent's own harness system.

Conclusion

The essence of Harness Engineering isn't a standardized technology — it's a distillation of engineering experience for using Agents effectively. The four principles are tightly interconnected: documentation serves as the standard, hooks handle automatic logging, relevant lessons are proactively injected before work begins, violation counts keep close watch, and entropy management maintains refinement.

The model provides the talent; Harness Engineering is responsible for steadily and repeatedly shaping that talent into exactly what you want. This is the key to taking an Agent from "functional" to "genuinely useful."