From Prompt to Harness: Deconstructing the Four Layers of Agent Engineering

A complete Agent task isn't just about getting the model to produce an answer — it's about ensuring information can flow in, tools run reliably, state carries over, and results can be verified. A talk on Bilibili about the evolution of Agent engineering broke this process down into four layers of capability: Prompt, Context, Harness, and AgentOS, using six questions to connect the full chain from single-turn Q&A to multi-Agent collaboration.

The core judgment of this talk is: the more complex the task, the less the model alone determines success or failure. How much information the context can capture, whether attention can stay focused, whether the current goal data can be accurately passed along, whether task state can be preserved across turns — ultimately, what needs to be solved are three engineering problems: capacity, precision, and continuity.

Why Complex Tasks Go Off the Rails

A single answer only requires judging whether the response is correct; a multi-step task must handle sequential dependencies; a continuous task must support interruption-resumption, result verification, and experience reuse. These three are not on the same order of complexity.

As tool results and historical messages accumulate, the proportion of context occupied by the current goal and valid evidence steadily declines. Once an erroneous operation occurs, the contamination gets repeatedly carried into subsequent steps, ultimately making it harder and harder for the model to grasp what truly matters. The talk gave an example: if you ask the model to regenerate 15 records that were already retrieved, omissions, tampering, and data changes can occur.

A more robust approach is to save the complete result as a reference, have the program accurately pass it to subsequent steps, and let the model focus on understanding and judgment. This leads to the first layer of Agent engineering: Prompt.

The Boundaries of Prompt Capability

Prompts can gradually evolve from simple role design into structural requirements and attention cues, but no matter how clearly the rules are written, they cannot replace a state management mechanism. According to the talk, an executable project specification must at minimum address four things: what the most important deliverable is, which operations are permitted, where to obtain relevant materials, how to verify results, and how to hand off unfinished work.

The problem is that Prompt-driven tasks typically can only run straight through to the end, and any step failing may force the entire process to restart. There is no persistent state, no control over context growth, and no deterministic validation of tool results. This is the ceiling of Prompt capability.

Context Window is a key concept for understanding this capability ceiling. It refers to the maximum number of tokens a model can process simultaneously in a single inference, including system prompts, historical messages, tool return results, and current input. The larger the window, the more information it can hold — but attention is not linearly distributed. Content located in the middle of the window tends to be more easily overlooked, which is the "lost in the middle" phenomenon. Therefore, simply expanding the window doesn't solve information overload — it may actually dilute the weight of critical signals. The truly effective approach is to prioritize and dynamically trim what enters the window, placing the current goal, necessary dependencies, and key evidence where the model's attention is most concentrated, while converting other details into retrievable references. Context-layer engineering is essentially about making trade-offs within a finite window, not simply pursuing capacity expansion.

Context: Precise Control Over Information Organization

At the Context layer, engineering means filtering the current goal and necessary dependencies before execution, organizing large results and key findings during execution, and retrieving evidence through references when details are needed. What it pursues is not more information but the right amount at each step.

Information bloat from different sources must be handled separately. Large results should be saved first, then generate parseable reference descriptions, boundaries, and summaries; structured handoffs for task continuation should read relevant fragments on demand when concrete evidence is needed. This controls context while preserving full traceability.

The talk specifically emphasized that effective conversation compression cannot just summarize what was discussed — it must also record where the task stands, what the next step depends on, what blockers were encountered, and where key evidence is stored. Only by forming a structured handoff can the next round of execution truly pick up where it left off. Information during execution is divided into three categories: State holds parameters and step outputs, Working memory holds the current goal and key findings, and Transcript retains only recent exchanges.

Harness: Making Execution Checkable and Recoverable

Even with information well managed, execution can still fail. Harness is responsible for recording where task execution stands, checking whether tool results meet the goal, and deciding whether to continue, retry, cancel, or hand off after a timeout, interruption, or failure. It turns the model's suggested actions into a manageable execution process.

Rather than constantly fixing errors after they occur, it's better to make the correct path more easily determinable by the program at the design stage. Key data should have tools return explicit states, require the model to promptly record findings, and retain relevant evidence whenever the current step involves it.

A single task consists of multiple executions and returns. A complete task can advance continuously through Plan, Execute, and Optimize, and each execution step internally goes through a small loop of Thought, Action, and Observation. A page disconnect doesn't have to mean the task stops — frontend information, message push, event reassembly, and backend execution should be separated from each other. Even if the page disconnects, the task can continue running and save its process. After reconnecting, missing progress and results are recovered based on the cursor, execution continues from the point of failure, and duplicate operations must be prevented.

The Thought-Action-Observation loop is the core structure of the ReAct paradigm: the model first produces reasoning (Thought), based on which it selects and invokes tools (Action), then receives the observation results returned by the tools (Observation), and this repeats until the task is complete. This loop allows the model to dynamically adjust strategies based on environmental feedback, but it also introduces two engineering challenges. First, the loop may fall into deadlock or ineffective retries, requiring Harness to set maximum step counts and failure criteria. Second, the intermediate state of each step must be persisted, otherwise once the process is interrupted, the entire previous reasoning chain is lost. Harness records input-output snapshots of each loop iteration, allowing tasks to resume from any breakpoint rather than restarting the entire process. Understanding this loop structure is understanding why the execution layer must exist independently of a single session.

AgentOS: From Single Agent to Collaborative Systems

As task scale expands, the system needs to simultaneously manage execution environments, runtime systems, memory and semantic cognition, scheduling, and experience governance. The talk recommends clearly dividing responsibilities at each layer first, then selecting necessary components — small projects don't need to build a full platform from the start.

Memory management needs to distinguish types. Industry rules, business knowledge, personal preferences, and current work state belong to different types of memory and should be stored separately. Even after memory has been recalled, permissions, scope of applicability, and conflict relationships must continue to be checked.

There is a clear verification chain for experience accumulation: after task completion, first record execution conditions, process, and results; then extract candidate experiences with scope of applicability and evidence; finally validate them in new tasks. Only experiences that withstand repeated verification are worth writing to long-term memory. Execution, learning, and governance are three different things — the execution loop cares about whether this task was completed, the learning loop cares about whether failure experiences can improve the next attempt, and the governance loop cares about whether new rules can be safely propagated.

Common scheduling patterns in multi-Agent collaboration include centralized orchestration and decentralized negotiation. In the centralized mode, an Orchestrator Agent is responsible for decomposing tasks and dispatching them to execution-type Agents, suitable for scenarios with clear processes and well-defined responsibilities. In the decentralized mode, Agents exchange information through a shared blackboard or message bus, offering greater flexibility but also higher coordination costs. Regardless of the mode, three problems must be solved: task ownership, result aggregation, and conflict resolution. If shared memory lacks permission layering, an erroneous experience written by one Agent can contaminate the entire system, so writing to long-term memory must go through a verification chain and scope annotation. The engineering difficulty of collaborative scheduling is not in getting Agents to communicate with each other, but in getting them to fulfill their roles within boundaries without producing cascading errors.

Where to Start Building

The practical path offered by this talk is: when output deviates from requirements, check the Prompt layer; when tasks break down, check the Context layer; when data transfer errors occur or processes cannot be recovered, check the Harness layer; when multiple Agents wait on each other, check the collaborative scheduling layer. First identify which layer the fault belongs to, then supplement the corresponding mechanism.

An Agent truly worth building doesn't just occasionally complete a task — it can continuously deliver within clear boundaries. Prompt is responsible for stating requirements clearly, Context for organizing information well, Harness for landing execution reliably, and AgentOS for managing collaboration at a larger scale. Turning "can do it once" into "can deliver repeatedly" — that is the meaning of Agent engineering.

One-line Summary

Agent engineering requires completing four layers of capability — Prompt, Context, Harness, and AgentOS — to solve capacity, precision, and continuity problems

Paragraph Summary

The article deconstructs Agent engineering into four layers of capability — Prompt, Context, Harness, and AgentOS — pointing out that the more complex the task, the less the model alone determines success or failure; rather, it comes down to three engineering problems: capacity, precision, and continuity. The Prompt layer is limited by the lack of persistent state and deterministic validation. The Context layer controls information bloat by saving large results as references and using structured handoffs. The Harness layer is responsible for making execution checkable and recoverable — tasks continue even when the page disconnects, and progress is recovered by cursor upon reconnection. The AgentOS layer manages multi-Agent collaboration, memory classification, and experience governance. The article emphasizes that experience accumulation requires three independent chains — execution, learning, and governance — and only repeatedly verified experiences are worth writing to long-term memory, and provides a practical path for locating the corresponding layer based on fault symptoms.