Harness Engineering: The Third Phase of AI Coding and a Practical Guide to agents.md

What a Research Paper Reveals About AI Coding

A paper titled Harness Engineering for Agentic AI Coding Tool has been generating significant buzz in the AI coding community. The research team systematically scanned 2,853 GitHub repositories — not randomly selected, but carefully filtered engineering-grade projects: those with multiple contributors, active licenses, and sustained activity. This "let real open-source data speak" approach follows the classic paradigm of Empirical Software Engineering, mining large-scale public GitHub repositories to analyze real-world development practices. By excluding personal experiments and zombie repos, the 2,853-repository sample is large enough to support statistically significant analysis, giving its conclusions strong industry reference value.

The scan results were surprising: 85.5% of AI Skills configurations are empty shells — defined but never executed. But another data point is even more telling — 493 repositories use only a single file, agents.md, with no vendor lock-in, no tool dependency. Developers voted with their feet and organically converged on their own configuration standard.

This is the starting point of Harness Engineering.

Single Q&A doesn't require configuration

What Is Harness Engineering?

Three Phases of AI Coding Paradigms

If you're hearing "Harness Engineering" for the first time, you're almost certainly familiar with its predecessors. AI coding practices have evolved through three phases:

Phase 1: Prompt Engineering — The core is "writing prompts": describe the problem clearly.
Phase 2: Context Engineering — The core is "designing context": give the AI the right background information.
Phase 3: Harness Engineering — Place the AI inside a structured framework that explicitly defines what it can and can't do, how to find files, how to run tests, and how to recover from errors.

Context Engineering has been one of the most important conceptual breakthroughs in AI engineering practice since 2024. Its core insight is that the quality of a large language model's output depends largely on the quality of the input context — not the model's capability ceiling. Context engineering focuses on systematically constructing inputs, including System Prompts, conversation history, externally injected documents via RAG (Retrieval-Augmented Generation), Tool Schemas, and structured Few-shot Examples. Prompt Engineering, by comparison, focuses on how to phrase a single question; Harness Engineering takes context management a step further — upgrading it from "manually constructed each time" to a "configure once, continuously effective" engineering system, giving AI consistent, structured context throughout the entire project lifecycle.

This reflects a profound shift in the developer's role: from "someone who writes code" to "someone who designs AI behavior."

Two Core Concepts: Agent and Tool

Understanding Harness Engineering requires clarifying two concepts the paper explicitly defines:

Agent: Goal-driven. Give it an objective, and it autonomously breaks down steps, selects tools, executes, adjusts, and re-executes — forming an autonomous loop.
Tool: Bounded functionality. A deterministic capability unit, like executing a command: called once, executed once, done.

Agentic AI represents a paradigm shift for large language models — from "Q&A assistant" to "autonomous executor." Unlike traditional single-turn dialogue models, Agentic AI has four core capabilities: Planning, Tool Use, Memory, and Self-correction. It can decompose a high-level goal into multiple subtasks, sequentially call external tools like file read/write, terminal commands, and network requests, and dynamically adjust strategy based on execution results — forming a closed-loop "sense-decide-act" cycle. Technically, this architecture relies on reasoning frameworks like ReAct (Reasoning + Acting), enabling the model to generate a traceable chain of thought before each decision.

The key distinction is: Agents are autonomous; Tools are passive. Precisely because Agents are autonomous, they need boundaries set for them — and that is the core value of Harness configuration.

Single Q&A vs. Continuous Execution

The traditional Copilot model works like this: write a Prompt, get a snippet of code, one conversation, one answer, done. The new-generation Agent model is completely different: write a configuration file, start a loop — read files, run tests, find issues, self-correct, run tests again, until the task is complete.

A single Q&A doesn't need configuration, but continuous execution must have boundaries. This is the core problem Harness Engineering solves.

Eight Configuration Mechanisms Explained

The paper systematically catalogues eight configuration mechanisms for controlling AI behavior, forming the complete Harness Engineering toolkit.

1. Context Files

This is the most fundamental and most important mechanism, with three core files:

agents.md: A cross-tool universal standard, recognized by Claude, Copilot, Cursor, and others alike.
CLAUDE.md: A configuration file specific to Claude Code.
GEMINI.md: A configuration file specific to Gemini.

These files serve as a project specification for the AI: what the project architecture looks like, how to build it, how to test it, and what the coding standards are.

2. Skills

Skills use a folder structure: SKILL.md (core instructions) + a Scripts directory (executable scripts) + a References directory (reference documentation) + an Assets directory (static resources). Bundled together, this forms a reusable capability unit. For example, a code review skill: instructions go in SKILL.md, review scripts go in Scripts, review standards go in References.

Yet 85.5% of Skills are "documentation only, no scripts" empty shells. This phenomenon has deep roots in software engineering. From a Cognitive Load perspective, building a complete skill package requires developers to simultaneously master multiple knowledge domains: abstracting business logic, writing shell scripts, handling cross-platform compatibility, and designing AI instructions — a composite skill threshold that is simply too high for most backend or frontend developers. This also reflects the "configuration as documentation" inertia trap: writing Markdown has almost zero execution cost, but writing runnable scripts requires debugging and maintenance investment. It's the same classic problem as low adoption of Test-Driven Development (TDD) or perpetually outdated API documentation.

3. Subagents

An advanced pattern: a primary Agent can dispatch multiple sub-Agents, each with independent context, tools, and state, executing in parallel without interfering with each other. For example, while the primary Agent writes code, it dispatches one sub-Agent to run tests and another to review documentation, then aggregates the results — essentially decomposing complex tasks and distributing them to specialized execution units.

4–5. Commands and Rules

Commands: Define command aliases — for example, review triggers a code review workflow, deploy triggers a deployment workflow.
Rules: Hard constraints — for example, "do not delete the production database," "do not commit untested code," "must follow PEP8 standards" — essentially putting a leash on the AI.

6–7. Settings and Hooks

Settings: Runtime parameters such as which model to use, timeout duration, and log level.
Hooks: Lifecycle triggers that fire automatically at specific moments — run tests before a commit, auto-deploy after passing, send a notification when a file is modified. This makes AI behavior predictable, auditable, and traceable.

Hooks are a classic software design pattern, widely used in Git, CI/CD pipelines (like GitHub Actions), and frontend frameworks (like React's useEffect). Their core idea is "automatically trigger predefined logic when a specific event occurs," transforming manual intervention into systematic automated responses. In AI coding scenarios, Hooks are especially valuable: they turn AI behavior from an "unpredictable black box" into an "auditable pipeline." The near-zero adoption of Hooks today also means most current AI coding practices are still at the "human supervision" stage — a considerable distance from truly autonomous execution.

8. MCP (Model Context Protocol)

The most cutting-edge mechanism, addressing how AI accesses external data — databases, APIs, Jira, Slack, GitHub, and other systems that aren't in the codebase but are often needed for AI tasks.

MCP (Model Context Protocol) was officially open-sourced by Anthropic in November 2024 and is one of the most strategically significant standardization protocols in the current AI tooling ecosystem. MCP uses a client-server architecture: the AI model acts as the client, various external systems (databases, APIs, file systems, etc.) act as MCP servers, and both communicate via a standardized JSON-RPC protocol. MCP defines three core capability types: Resources (structured data reading), Tools (executable operation calls), and Prompts (preset prompt templates). Developers only need to implement an MCP server once for a given system (like PostgreSQL or Jira), and all MCP-compatible AI tools can call it directly — fundamentally solving the fragmentation problem of connecting AI tools to the outside world. MCP acts as a standardized bridge, letting AI securely connect to external systems for read/write operations, without binding to any specific tool.

The Hard Data: The Gap Between Ideal and Reality

The data analysis section of the paper is the most striking, revealing the true state of current AI coding configuration practices.

Most Teams Haven't Started Multi-Tool Collaboration

85.4% of repositories configure only a single tool, indicating that multi-tool collaboration has yet to become mainstream. Among these, Claude Code accounts for 45.2%, making it the most popular AI coding tool today, with Cursor and Copilot close behind. But there is almost no integration between tools — they mostly operate in isolation.

The Organic Rise of agents.md

Timeline analysis reveals an interesting trend: CLAUDE.md typically appears first (because Claude Code was the earliest to provide configuration capabilities), and developers later add agents.md. The reason is simple — agents.md is universal; it can be read regardless of which tool you switch to. Ultimately, 493 repositories use only agents.md, without any tool-specific configuration.

This reflects the deep-seated tension of "open standards vs. vendor lock-in" in the AI tooling ecosystem. Vendor lock-in is a classic risk in enterprise technology decisions — once you're deeply bound to a proprietary format, migration costs grow exponentially with configuration complexity. Cloud computing went through the same evolutionary path: from each vendor's private APIs, to the rise of open standards like OpenStack and Kubernetes. The organic popularity of agents.md signals a similar standardization trend — developers are expressing, through their actions, a strong preference for portability and tool-agnosticism. This is a bottom-up standardization movement.

The most interesting finding

Advanced Features Collectively Abandoned

The gap between ideal and reality is stark:

85.5% of Skills are empty shells — documentation without scripts, defined but never executed.
Zero adoption rate for Subagents' persistent memory features.
Hooks and MCP are almost entirely unused.

The root cause is high configuration cost: writing a simple SKILL.md is easy, but building a complete skill package with scripts, references, and assets has too high a barrier. So most people take the path of least resistance — write documentation, skip the scripts.

Comparing Configuration Styles Across Three Major Tools

The paper compares the configuration ecosystems of the three mainstream tools, revealing distinct stylistic differences:

Claude Code: The most complex, covering all eight mechanisms, with the deepest user configuration depth.
Cursor: Emphasizes Rules and Commands; the .cursorrules file is its signature configuration.
Copilot: The simplest — relies almost exclusively on Context Files, rarely extending to other mechanisms.

Three tools, three styles, three ecosystems — catering to different teams' configuration preferences. Each tool's proprietary configuration capabilities must offer a significant advantage over the universal standard, or it will struggle to retain engineering teams with long-term planning horizons.

If you use both Claude and Cursor

Three Actionable Recommendations

The paper closes with three actionable recommendations, especially relevant for developers just getting started:

Start with agents.md. Regardless of what tool you're currently using, write an agents.md first — put in the project architecture, build commands, and testing standards. This is the lowest-cost starting point.
Maintain a shared core configuration in multi-tool environments. If you use both Claude and Cursor, have them read the same agents.md to avoid maintaining duplicate configurations.
Use Skills to package repetitive workflows. If your team has a fixed code review or release process, package it into a Skill for reuse — reducing repeated configuration overhead.

Configuration Files Will Become Core Team Assets

The essence of Harness Engineering is managing AI behavior through configuration files — not guessing via prompts, not guiding through conversation, but writing config, defining rules, and letting AI work within clear boundaries.

More importantly, this isn't just an individual practice — it's the foundational infrastructure for team collaboration. Imagine this scenario: a team shares a single agents.md, and every modification goes through discussion and a code review process — because it determines how the AI understands the project, how it executes tasks, and what standards it follows. Configuration files will become indispensable core team assets, on par with code itself.

The trend is clear: agents.md is becoming the de facto standard, and developers are voting with their feet — choosing universal standards and rejecting vendor lock-in. This evolutionary path mirrors Kubernetes's rise in container orchestration — the community's organic convergence is often more durable than any top-down design. If your AI coding assistant isn't delivering the results you want, the problem may not be the tool — it may be that your configuration approach hasn't kept up. Consider starting today: write an agents.md for your project, and take your first step into Harness Engineering.