SkillPlumber: A Coverage-Guided Framework for Self-Evolving Agent Skills

SkillPlumber uses instruction-level coverage to distinguish instruction errors from execution omissions, enabling agents to precisely evolve skills where needed.
The article introduces the SkillPlumber framework. To address the attribution difficulty in existing agent skill self-evolution methods—which cannot distinguish between "errors in the instructions themselves" and "the agent failing to follow instructions"—it proposes instruction-level coverage as the key. Coverage marks each instruction as followed, not followed, or not covered, thereby refining failure attribution to the instruction level. The framework has two roles: a root-cause analysis agent that translates coverage labels into specific revision questions and verifies them in a closed loop, and a maintainer that distills a shared strategy library across tasks. Experiments show that across three benchmarks and three executors, task resolution reaches 64.7%-70.5%, a 25.6-30.4 percentage point improvement over initial skills and 8.7-14.0 points over the strongest baseline; ablation shows that removing the coverage mechanism reduces recovered failure cases by 20.8%. The article also discusses limitations and future directions.
Introduction: When Agent Skills Learn to "Self-Evolve"
An agent's skill is essentially procedural knowledge packaged as natural-language instructions, and it only truly works when the instructions are correct and the agent executes them correctly. In recent years, having agents revise their skills based on their own run logs has become a research hotspot, but existing methods typically judge based on overall run outcomes—either looking at task success or failure, or at where errors occurred—yet they cannot distinguish whether a failure stems from an error in the instructions themselves or from the agent failing to follow correct instructions. This fundamental attribution difficulty causes self-evolution to often iterate in the wrong direction.

According to the approach proposed in this paper, instruction-level coverage can be the key to breaking the deadlock: it marks each instruction in a run as "followed," "not followed," or "not covered," thereby separating the two types of failure causes. Based on this insight, the researchers built the SkillPlumber framework, enabling agents to evolve precisely in the areas where coverage reveals shortcomings.
Procedural knowledge is a concept contrasted with declarative knowledge; it refers to knowledge of "how to do something," rather than factual knowledge of "what something is." In cognitive science, it typically manifests as executable step sequences or rule sets, often difficult to fully articulate in language (i.e., tacit knowledge), and requires internalization through practice. In the agent context, skills encapsulate procedural knowledge in the form of natural-language instructions, making it readable and executable by models. The advantage of this encapsulation is interpretability and editability, but it also brings a hidden risk: the ambiguity of natural-language instructions combined with the uncertainty of model execution makes attributing "skill errors" difficult—which is exactly the core problem discussed in the article. Understanding the characteristics of procedural knowledge helps grasp why instruction-level diagnosis is closer to the essence of the problem than outcome-level feedback.
Blind Spots of Existing Self-Evolution Methods
Current mainstream agent skill self-evolution methods typically have the model examine the overall trajectory of a run and then revise the skill description accordingly. This approach seems reasonable but ignores the issue of attribution granularity.
Suppose a task fails: it could be that the instructions were written incorrectly, or that the instructions were fine but the agent skipped key steps during execution. Traditional methods cannot distinguish these two cases, so revision actions tend to "overshoot"—rewriting instructions that were already correct, or prescribing the wrong "medicine" for execution problems. As a result, skills do not actually improve through iteration and may instead introduce new inconsistencies.
Instruction-level coverage labels provide a fine-grained diagnostic signal: when an instruction is marked "not followed," it suggests the skill itself may be fine and the problem lies in execution; when an instruction is "not covered," it suggests the skill may be missing necessary steps. It is precisely this distinction that makes "where to evolve" evidence-based.
Attribution granularity refers to the smallest unit of analysis used when analyzing the cause of a failure. Coarse-grained attribution treats a task run as a single event and can only yield "success/failure" or "the approximate location where the error occurred"; fine-grained attribution drills down to the step, instruction, or even token level, attempting to pinpoint exactly which link went wrong. The finer the granularity, the more instructive the feedback signal, but the higher the acquisition cost. The credit assignment problem in machine learning shares the same origin: when a system performs poorly overall, how to distribute "responsibility" across components. Instruction-level coverage is essentially a fine-grained credit assignment signal obtainable manually; it bypasses the credit assignment challenge in end-to-end learning and directly tells the agent the execution status of each instruction in label form, giving revision actions a clear target.
Core Design of the SkillPlumber Framework
The overall idea of SkillPlumber is: coverage points out shortcomings, the agent revises skills targeting those shortcomings, and the revision effect is verified through subsequent runs. The framework contains two key roles.
Root-Cause Analysis Agent: Turning Labels into Revision Questions
A persistent root-cause analysis agent is responsible for receiving the coverage labels produced by each run and translating each label into a specific revision question. For example, a "not followed" label might correspond to "Is the instruction wording unclear, causing execution omissions?", while a "not covered" label might correspond to "Is the skill missing the necessary guidance to complete this step?". In this way, vague failure signals are translated into actionable improvement directions.
More critically, each revision is tested in the next round of runs, forming a closed loop. The root-cause analysis agent does not give modifications one time only; it continuously tracks whether revisions actually solve the problem.
Maintainer: Distilling a Shared Strategy Library Across Tasks
Another role, the "maintainer," is responsible for distilling investigation results across tasks into a shared strategy library. Revision experience from a single task is often local, but the maintainer abstracts this experience into more general strategies, so that skill evolution is no longer confined to a single task and can accumulate into reusable knowledge assets. This design gives the agent's self-evolution cross-task transfer capability.
Closed-loop verification is a key design pattern in autonomous agent systems: any modification is not a one-time action but must be tested in subsequent runs, with the results determining whether to keep, roll back, or further adjust. Compared with open-loop systems, the advantage of closed-loop systems is the ability to suppress the accumulation of erroneous modifications—if a revision causes performance to decline, the next round of runs will expose the problem and trigger further revision. In SkillPlumber, the root-cause analysis agent plays exactly the role of a closed-loop controller: it receives coverage labels as observation signals, outputs skill revisions as control actions, and uses the coverage of the new round of runs as feedback. This "observation–action–feedback" structure gives self-evolution stability and avoids the drift phenomenon of "the more it's changed, the messier it gets" in traditional methods.
Experimental Performance: The Gains from Coverage
According to the paper, SkillPlumber was evaluated on three benchmarks and three executors, with task resolution rates between 64.7% and 70.5%. Compared with initial skills, the improvement is 25.6 to 30.4 percentage points; compared with the strongest baseline method, the improvement is 8.7 to 14.0 percentage points.
Ablation experiments further validate the value of coverage: when using GPT-5.5 on SkillsBench, removing the coverage mechanism reduces recovered failure cases by 20.8%. This data shows that coverage guidance is not just a nice-to-have but one of the core sources of the framework's effectiveness.
You may not have noticed that these improvements were consistently observed across multiple executors, indicating that the method has some robustness to changes in the underlying model.
In-Depth Analysis: Why Coverage Is a Better Evolution Signal
From a methodological perspective, the value of SkillPlumber lies in advancing "skill revision" from coarse-grained outcome feedback to fine-grained process diagnosis. Traditional methods judge a run as a whole, essentially losing instruction-level information; coverage labels preserve the execution status of each instruction, enabling revision actions to target the right problem.
This approach also responds to a core contradiction in the field of agent self-evolution: the reliability of self-assessment. If an agent cannot accurately determine where it went wrong, self-evolution may become self-misleading. Coverage provides a relatively objective external signal, reducing the agent's subjective bias in attribution.
Another noteworthy design is the division of labor between the root-cause analysis agent and the maintainer. The former focuses on problem localization for a single run, while the latter is responsible for cross-task strategy distillation; together they balance local correction and global accumulation. This layered architecture has reference value for building sustainably evolving agent systems.
Self-assessment reliability is a fundamental constraint in agent self-evolution research. When a model acts as both "executor" and "judge," its judgment is susceptible to its own biases: it tends to consider the steps it executed as reasonable, thereby underestimating execution omissions; it may also misjudge the correctness of the instructions themselves due to overconfidence. Such bias is difficult to correct in the absence of external signals. The value of instruction-level coverage lies in providing an observation relatively independent of the model's subjective judgment—whether each instruction was followed can be objectively determined through mechanisms such as trajectory comparison, rather than relying on model introspection. This is in line with the idea of introducing external reward signals in reinforcement learning rather than relying solely on model self-judgment; essentially, it injects an "anchor" into the self-evolution system to prevent it from drifting in a self-referential loop.
Limitations and Outlook
The current framework relies on obtaining instruction-level coverage, which requires skills to be decomposable into trackable instruction units. For skills with ambiguous instruction boundaries or heavy reliance on implicit knowledge, the accuracy of coverage labeling may become a bottleneck. In addition, the paper has not yet fully discussed how the distillation effect of the shared strategy library scales with task size.
Nevertheless, the principle proposed by SkillPlumber—"evolve where coverage reveals insufficiency"—provides a clear and verifiable direction for agent skill self-evolution. Future work could explore more automated ways to obtain coverage, as well as the transferability of the strategy library across different domains.
Conclusion
The core contribution of SkillPlumber is not to make agents revise skills more frequently, but to make revisions happen where they are truly needed. By using instruction-level coverage to distinguish "instruction errors" from "execution omissions," the framework transforms self-evolution from blind overall feedback into targeted local repair. For researchers and developers concerned with agent reliability and continual learning, this approach deserves ongoing attention.
One-line Summary
SkillPlumber uses instruction-level coverage to distinguish instruction errors from execution omissions, enabling agents to precisely evolve skills where truly needed.
Related articles

Watching Movies in the Terminal: Building an SSH Streaming Service with Codex Multi-Agent Workflow
A creator uses Codex multi-agent workflow to build an SSH-accessible streaming service in the terminal, with planning, cloud async dev, AI-drawn scenes, and synced live channels.

AI Face Licensing: A New Micro-Drama Trend or a Hollywood Trap?
Actors are licensing their faces to micro-drama startups for AI digital doubles to perform. Popular in China, the model raises concerns in Hollywood over lost control and diluted bargaining power.

Amazon's First Digital Photo Frame Is 50% Off for Black Friday: Is the Alexa Frame Worth Buying?
Amazon's first digital photo frame, the Alexa Photo Frame, is nearly 50% off for Black Friday, with prices starting at $114. We analyze why this new product dropped in price just one month after launch, its positioning, who it's for, and buying advice.