Open-Sourcing an AI Agent Skill Tool That Failed Its Own Kill Rule

Summary: The developers pre-wrote a "kill rule" — if their structured skill-generation pipeline couldn't beat one-shot generation on trigger recall, they would publicly declare the hypothesis dead and change the product. The pipeline lost, 98.3% vs 91.7%. But they open-sourced Skill Forge anyway, because the skill-file failure mechanisms exposed during the experiment are the tool's real value.

---

Before the Experiment, They Wrote the Death Warrant

Before running the experiment, the team wrote down the rule that would kill their product. The subject under test was a pipeline for building skill files (SKILL.md) for AI agents: through requirements interviews, controlled generation, and mechanical audits, it produces structured instruction documents — telling the agent when to trigger, what to do, and what to never do.

The hypothesis seemed intuitive: skills that went through the structured pipeline should beat skills the model improvised one-shot. They pre-registered four predictions and one kill criterion — if the pipeline couldn't quantitatively beat one-shot generation, they would publicly admit it and change the product.

The results came in. One-shot generation beat the pipeline on trigger recall at 98.3% vs 91.7% (118 out of 120 vs 110), with overlapping 95% Wilson intervals. By the rule written to disk beforehand, the hypothesis was dead.

Yet the team still open-sourced Skill Forge (part of the wallaby-agent-rules repository, MIT + Commons Clause license). This seemingly contradictory move is precisely the most valuable part of the whole experiment. The kill rule tested only one claim: whether structured generation could beat improvisation at the routing layer. That claim died, but the experimental project around it produced a set of measurement findings about how skill files actually fail — each of which became a load-bearing design decision in the tool.

Preregistration is the key to this experimental methodology. Before running the data, researchers write down the hypothesis, predictions, evaluation metrics, and — most critically — the threshold that declares the hypothesis dead, then commit it to a file on disk. Its core value is cutting off the retreat path of "post-hoc narrative": without preregistration, the team could have seen 91.7% and simply redefined what counts as "beating," switched the metric from trigger recall to some other dimension, or just not published the results. The soul of preregistration is a kill rule — agreeing in advance on what result would make you concede. This team relied on exactly that rule to prevent self-deception once the data came in, and could only admit that "structured generation beats improvisation at the routing layer" was dead. This practice has traditions in machine learning, psychology, and medical clinical trials, but is rare in the context of individual developers building AI tools, because it requires you to emotionally accept that your product might fail.

What Died, and What the Corpse Taught Us

The Description Field Became a Stop Sign

The third round of experiments was a routing test, using the weak model DeepSeek^Ad V4.1-Flash, with 480 routing decisions, on-chip order shuffled by seed. The pipeline group ran the full flow; the one-shot group had the model write the skill file directly.

The pipeline lost, and all its failures failed in the same way: the model simply refused to trigger, never selecting the wrong skill. The problem lay in the description field — the generator kept writing boundary clauses into it, like "does not trigger on X" or "only when Y." This kind of negative scoping looks like good practice to humans, but to small models making relevance judgments, they are stop signs.

After completely removing boundary sentences from the description, this class of failure dropped from 10 to 0.

Wilson interval is a method for estimating confidence intervals for binomial proportions (like "got 118 of 120 right"), and compared to the more common normal approximation method, it performs more robustly with small samples and proportions near the boundaries (0 or 1). The "overlapping 95% Wilson intervals" mentioned in this article means the confidence intervals of the two groups — the pipeline group at 91.7% and the one-shot group at 98.3% — overlap, i.e., statistically you cannot reliably say these two are truly different, and the observed gap may just be sampling noise. The team repeatedly uses Wilson intervals to stay honest under small-sample conditions (ten to twenty per group): only when the gap is large enough that the intervals no longer overlap do they dare treat it as a real effect. This is also why they treat "the third-round failure" as a real difference outside the intervals, while treating several secondary findings as noise inside the intervals. Understanding Wilson intervals is the prerequisite for reading "what counts as losing" and "what's just noise" throughout the article.

Trap Questions Exposed the "Magnet Skill"

The fourth round turned to a harder domain: eight requirements in a real project domain, each skill having a near neighbor, with trap questions that sound like they belong to two skills at once. The kill rule triggered again, but the gap shrank from 6.6 percentage points to 2.7, and the failure mode changed — the model started picking the wrong near neighbor instead of refusing to trigger.

There was an internal ops skill that acted like a magnet pulling both groups, and trap questions hit it twice. The hand-tuned champion version answered perfectly, because its description carried a routing verdict line that directly named this confusion scenario. No generator had ever seen its near neighbor, so it couldn't write that sentence. Now the pipeline proactively asks during generation: which sibling skills can your skill not be confused with?

Routing here refers to the process by which an agent, upon receiving a user request, decides which skill file to invoke — essentially classifying the request for relevance. This judgment is often made by a "weak model" (like the DeepSeek V4.1-Flash mentioned in the article), because routing calls are frequent and cost-sensitive, and it's impossible to use the strongest model every time. Trigger recall measures: of all sentences that should have triggered a given skill, how many did the model actually trigger. It differs from precision — low recall means the skill is under-used (should trigger but doesn't), while low precision means it's misused (triggers when it shouldn't). The failure mode in this article is mainly a recall problem: the model doesn't pick the wrong skill, it simply doesn't trigger at all. Understanding these two concepts is what makes clear why "negative clauses in the description" are fatal — it's not that they bias the routing judgment slightly, it's that they directly trigger a "do not activate" decision.

Footnotes Always Score Zero

The first conclusion that survived two rounds of experiments: the mechanical audit — that boring structural health check — caught the one-shot group fabricating pitfalls that never happened and citing prices that didn't exist. The pipeline group had not a single red flag, not a single fabricated detail across all generated files.

The second conclusion, called by the harshest external reviewer the most valuable insight in the document: every forge skill ended with a maintenance instruction — "if something goes wrong during operation, record it to the failure ledger" — placed as a footnote at the bottom of the file. Twenty executions, two models, not a single trigger. Changing the wording didn't help; making it a separate section didn't help. The fix was to move the instruction into the numbered task steps, as the final step of the run. The same sentence, in a new position: the weak model's feedback-record score rose from 0.00 to 1.80 (out of 2).

The third preregistration also failed, and is equally worth naming: when subsequently testing the "ideal form" hypothesis, three of four predictions were wrong. The tool available for download today carries the scars of three rounds of experiments, not just victories.

After the Funeral, What Is This Tool Anyway

The team rewrote the product claim to match the evidence: Skill Forge doesn't produce capability-equal models, it keeps quality requirements consistent across models — the same triggers, red lines, acceptance checks, and failure records, applied to every executor. The center of gravity shifted from "creation" to "lifecycle": health checks, failure ledgers, controlled revision.

In practice it's three things:

1. Derivation protocol: An interview skeleton extracts requirements from what the user has already said, with each answer falling into one of three buckets — "user confirmed / safely derived / open and critical" — and only the third bucket can become a question. A high-risk checklist forces the user to explicitly confirm anything involving production deployment, deletion, money, credentials, or public release.
2. Generation spec with a birth certificate: Generated skills come with trigger test pairs — sentences that should trigger and sentences that shouldn't — as a regression baseline that can be rerun after any edit.
3. Maintenance loop: the failure ledger the skill writes into itself (thanks to the embedded closing step), seven health checks, and sentence-by-sentence four-verdict optimization: prune, sink to a reference file, demote to on-demand reading, or keep.

Each item exists because a measurement put it there.

Five Design Decisions, Five Receipts

1. The description only carries positive trigger conditions

The generator now treats the frontmatter description as a restricted zone: only positive trigger phrases, written the way users actually speak. Input-shape restrictions like "plaintext only" or "only when X" are completely banned, because the router executes them as "do not trigger" rules and recall falls off a cliff.

Boundaries aren't banned, just relocated: abstract negations are replaced with near-neighbor verdict lines — specific sentences naming specific confusion scenarios. Abstract boundaries are invisible at routing time, but named verdicts can match trap questions word for word.

2. Failure records are the last numbered step, never a footnote

The "twenty times, zero triggers" finding is now structural: in every generated skill, the failure-record instruction is the last numbered step in the task flow — not a tail note, not a separate section, not a "remember to" at the bottom.

This conclusion generalizes beyond the tool and may be the most portable insight of this release: agents execute task steps, not footnotes. If your AGENTS.md or system prompt ends with "remember to update the changelog," you've written a footnote, and footnotes score zero. Move it into the work.

3. Red lines are specific, checkable, never blanket

Both wording rules were measured on the hard task set. Every red line must be specifically checkable: "never guess the amount field," not "never make mistakes." Blanket shutdown clauses are banned — "if conditions are not fully met, stop completely" reads as caution to a human, but strong models execute it as righteous shutdown, refusing tasks that could have been completed.

The same scaffolding raised the weak model's execution quality from 0.88 to 2.0 (across four scoring dimensions), yet pushed the strong model toward over-cautious shutdown (all four dimensions dropped). Structure helps the weak and can also shackle the strong. So the pipeline routes along two axes: model strength + boundary ambiguity, deciding whether the skill ships as a full scaffold or a thin shell.

4. Size is a hard budget, counted in bytes

A skill file that naturally bloated to 9.2KB made the weak model return empty responses six times in a row. Isolating the size variable produced a clean gradient: below 6KB, eight runs with zero burn-through; 6–9KB, about 25% empty responses; above 9KB, about 50%.

The tool treats 6144 bytes as the safe zone, enforced precisely by byte. When exceeded, it routes to a layered form: the body keeps only high-frequency rules and a scenario routing table, with scenario details sunk into references/ for on-demand reading. Ceremonial content (pitfall placeholders, feedback hooks, changelog) has a hard budget of three lines — every line of ceremony is an instruction line the model won't read.

5. Audits are mechanical, and the ledger has an idle alarm

The health check is deliberately boring: is the description trigger-style wording? Is the size in the safe zone? Are there hardcoded versions, dates, prices? Are commands using full paths? Are there red lines when content touches shell, network, files, or money? Is the pitfall section real? Is the ceremony budget being kept? Boring is exactly the point — it was this check that caught the one-shot group's fabricated pitfalls and prices.

The failure ledger appends one line of JSON per failure: date, scenario, what broke, with a secret-redaction rule attached. Because "twenty times, zero triggers" taught the team: silence usually means the hook isn't running. So the ledger has an idle rule — history present but 14 days with no new records is a signal, not comfort: either there really are no failures (possible), or no one is recording (more likely), and the optimization flow tells you which to check first.

Three Forms, One Protocol

The same protocol ships in three ways, covering everyone from those running an MCP server to those who just want to paste something into a chat.

• MCP server (extras/skill-forge/mcp/): zero dependencies, Python 3 standard library only, stdio transport. Three tools: skill_forge to create, skill_optimize for health checks plus rewriting, skill_feedback for ledger writes. It automates all the mechanical parts: chip scanning, redacted ledger writes, idle warnings, byte-precise size metering, seven-item reports.
• Prompt pack (extras/skill-forge/prompt-pack/): the same protocol as Markdown files, to paste into any agent conversation. The five automations become a manual checklist. No installation, no server.
• Skill form (extras/skill-forge/skill-form/): the protocol itself packaged as SKILL.md, for Hermes and any tool following that convention. Generated at release time by a sync script from the prompt pack, ensuring a single source of truth and no drift between forms.

The MCP quick start is three lines of config in any stdio client (Python3 + absolute path to the script). The repo ships with a 78-assertion smoke suite that spins up a real server instance against a temporary HOME to check the protocol surface: python3 mcp/smoke_test.py.

Honest Limitations

The same discipline that killed the original claim also applies to what they shipped:

• Small samples, most rounds ten to twenty per group. They publish Wilson intervals and treat near-ties as noise. The third-round failure was outside the interval; several secondary findings were inside it, and they say so honestly.
• The main work was done by one model family, with the judging model correlated with the weak executor. Follow-up behavior checks ran other models (including having Claude review the forge spec and send back test tooling), but the core measurements still describe one family's behavior.
• Generation and rewriting semantics still depend on the executing model. A weak model may not follow a long spec, and no scaffolding can eliminate that. The tool raises the consistency of requirements; the ceiling remains where the model puts it.
• The entire experimental project behind three blog posts cost ¥10.6 in API fees, about $1.5. That's the honest scale of the evidence: real, measurable, and small.

Why Open Source, Why Now

The team built their agent infrastructure in a specific order: hit real problems while running their own agent, build the fix, run it in production, publish what happened, then productize. The memory side was previously released as wallaby-agent-rules — a pure-file memory system where every fact carries a receipt. Skill Forge is the same governance philosophy one layer up.

The memory system governs what your agent remembers; this governs what it does. Both are rulebooks they run themselves, published as-is, failure logs included.

License is MIT + Commons Clause: free for personal projects, research, and internal organizational use; commercial license required to sell it as a hosted or bundled product.

For anyone building agent skills, or who has been burned the same way by the same skill over and over, the team invites you to run through it. The prompt pack costs no more than a paste. "No claims about what should work. Only what we measured."

Key Takeaways

1. The kill rule killed a claim, not the tool. The pipeline lost to one-shot generation on trigger recall (98.3% vs 91.7%), but the failure mechanisms the experiment exposed became the tool's real skeleton.
2. Negative boundaries in the description are stop signs. Small models treat "does not trigger on X" as a do-not-trigger instruction; rewriting boundaries as near-neighbor verdict lines can match trap questions word for word.
3. Footnotes always score zero. A failure-record instruction placed as a footnote at the bottom scored zero triggers in twenty executions; after moving it into the numbered task steps, the weak model's score rose from 0.00 to 1.80. Agents execute steps, not footnotes.
4. Structure helps the weak and can also shackle the strong. The same scaffolding raised the weak model's execution quality from 0.88 to 2.0, yet made the strong model fully retreat on ambiguous-boundary tasks — so route along two axes: model strength and boundary ambiguity.
5. Size is a hard budget. Below 6KB, zero burn-through; above 9KB, about half empty responses; ceremonial content has a hard three-line budget. The entire experimental project cost only about $1.5 in API fees — real, measurable, and small.

Background Notes

Scaffolding refers to the structured support wrapped around skill execution — such as numbered task steps, health checks, red-line checklists, failure-ledger hooks, and so on. They don't produce capability themselves, but they constrain and guide how the model uses capability. The "full scaffold" and "thin shell form" that recur throughout the article are two versions of the same protocol trimmed to need: the full version adds structure and discipline for weak models, the thin shell version sets fewer constraints for strong models. Inverse scaling is the more general phenomenon — certain prompts, constraints, or scaffolding help weak models but harm strong ones, because they push a strong model capable of flexible judgment toward mechanical compliance or over-caution. Design decision #3 in this article is exactly an instance of this phenomenon: the same scaffolding raised the weak model's execution quality from 0.88 to 2.0, yet made the strong model fully retreat on ambiguous-boundary tasks, so the tool decides which form to ship based on the two axes of "model strength × boundary ambiguity."

One-line Summary

A skill-generation pipeline with a preregistered kill rule lost to one-shot generation, yet found real value in the failure mechanisms it exposed.

Paragraph Summary

The team pre-wrote a kill rule: if their structured skill-generation pipeline couldn't beat one-shot generation on trigger recall, they would declare the hypothesis dead. The pipeline lost, 91.7% to one-shot's 98.3%. But they still open-sourced Skill Forge, because the failure mechanisms revealed during the experiment are the tool's real skeleton. Negative boundaries in the description were treated as stop signs by small models; a failure-record instruction placed as a footnote scored zero triggers in twenty executions, and moving it into numbered steps raised the weak model's score from 0.00 to 1.80; the same scaffolding improved weak models but shackled strong ones, requiring two-axis routing by model strength and boundary ambiguity; below 6KB saw zero burn-through, above 9KB about half empty responses. The whole project cost only about $1.5 in API fees — real, measurable, and small.