Skill Forge
An open-source pipeline tool that builds structured skill files (SKILL.md) for AI agents. Through requirements interviews, controlled generation, and mechanical auditing, it produces instruction documents that tell the agent when to trigger, what to do, and what never to do.
Timeline (last 90 days)
Skill Forge is a pipeline for building skill files (SKILL.md) for AI agents; through requirements interviews, controlled generation and mechanical audits it produces structured instruction documents, belongs to the wallaby-agent-rules repository, and uses the MIT + Commons Clause license.
On trigger recall, one-shot generation beat the structured pipeline 98.3% (118/120) versus 91.7% (110/120), with overlapping 95% Wilson intervals.
In the third round of routing tests, using the weak model DeepSeek V4.1-Flash, across 480 routing judgments, all failures in the pipeline group manifested as the model refusing to trigger rather than choosing the wrong skill; after removing the boundary negation clause from the description, such failures dropped from 10 to 0.
In the fourth round of experiments, trap questions narrowed the gap from 6.6 percentage points in the third round to 2.7 percentage points, and the failure mode became the model choosing the wrong neighboring skill rather than refusing to trigger.
The hand-tuned champion version's skill contains a routing decision line that directly names the confusing scenario, and no generator had ever seen its neighbor, so it could not write that decision line.
Mechanical audits can catch the improvised group fabricating pitfalls that never happened and citing prices that do not exist; the pipeline group had no red flags or fabricated details in any generated file.
After moving the failure-logging instruction from a footnote into a numbered task step, the weak model's feedback-logging score rose from 0.00 to 1.80 (out of 2), and across twenty runs neither model triggered the instruction in the footnote version.
The same scaffold raised the weak model's execution quality from 0.88 to 2.0 (across four scoring dimensions), yet pushed the strong model toward overcautious shutdown (all four dimensions dropped).
Skill file size is under a hard budget measured in bytes: below 6KB, zero burnouts across eight runs; 6–9KB, about 25% empty responses; above 9KB, about 50%; one skill file that naturally ballooned to 9.2KB made the weak model return empty responses six times in a row.
Skill Forge does not manufacture models of equal capability, but rather makes quality requirements consistent across different models, shifting the focus from authoring to lifecycle (health checks, failure ledgers, controlled revisions).
2 more timeline events
All Facts (12)
Skill Forge is a pipeline for building skill files (SKILL.md) for AI agents; through requirements interviews, controlled generation and mechanical audits it produces structured instruction documents, belongs to the wallaby-agent-rules repository, and uses the MIT + Commons Clause license.
50%UnverifiedOn trigger recall, one-shot generation beat the structured pipeline 98.3% (118/120) versus 91.7% (110/120), with overlapping 95% Wilson intervals.
50%UnverifiedIn the third round of routing tests, using the weak model DeepSeek V4.1-Flash, across 480 routing judgments, all failures in the pipeline group manifested as the model refusing to trigger rather than choosing the wrong skill; after removing the boundary negation clause from the description, such failures dropped from 10 to 0.
50%UnverifiedIn the fourth round of experiments, trap questions narrowed the gap from 6.6 percentage points in the third round to 2.7 percentage points, and the failure mode became the model choosing the wrong neighboring skill rather than refusing to trigger.
50%UnverifiedThe hand-tuned champion version's skill contains a routing decision line that directly names the confusing scenario, and no generator had ever seen its neighbor, so it could not write that decision line.
50%UnverifiedMechanical audits can catch the improvised group fabricating pitfalls that never happened and citing prices that do not exist; the pipeline group had no red flags or fabricated details in any generated file.
50%UnverifiedAfter moving the failure-logging instruction from a footnote into a numbered task step, the weak model's feedback-logging score rose from 0.00 to 1.80 (out of 2), and across twenty runs neither model triggered the instruction in the footnote version.
50%UnverifiedThe same scaffold raised the weak model's execution quality from 0.88 to 2.0 (across four scoring dimensions), yet pushed the strong model toward overcautious shutdown (all four dimensions dropped).
50%UnverifiedSkill file size is under a hard budget measured in bytes: below 6KB, zero burnouts across eight runs; 6–9KB, about 25% empty responses; above 9KB, about 50%; one skill file that naturally ballooned to 9.2KB made the weak model return empty responses six times in a row.
50%UnverifiedSkill Forge does not manufacture models of equal capability, but rather makes quality requirements consistent across different models, shifting the focus from authoring to lifecycle (health checks, failure ledgers, controlled revisions).
50%UnverifiedIn practice Skill Forge comprises three parts: a derivation protocol (an interview skeleton extracts requirements, and a high-risk checklist forces explicit confirmation), generation specifications with a birth certificate (triggering test pairs serve as a regression baseline), and a maintenance loop (failure ledger, seven health checks, sentence-by-sentence four-verdict optimization).
50%UnverifiedBlanket shutdown clauses (such as "if conditions are not fully met, stop completely") read as cautious to humans, but strong models execute them as shutdowns of justice, refusing tasks that could have been completed.
50%