Tested: When Skill Files Exceed 6KB, Model Output Failure Rates Double

Title

Tested: When Skill Files Exceed 6KB, Model Output Failure Rates Double

Meta Description

Based on 28 test runs: once a skill file exceeds 6KB, the model's empty output rate reaches 25%, and a naturally bloated 9.2KB file hits 50%. This article breaks down the experimental data, the ~1,500 token mechanism explanation, and three practical rules for keeping skill files under 6KB.

Article Content

There's a rarely discussed upper limit for writing skill files for coding agents (whether called SKILL.md, AGENTS.md, or rules): 6KB. This is the conclusion a team reached after running 28 experiments, and this article breaks down its data, mechanism, and boundaries.

The Starting Point: A File That "Isn't a Bug"

The experiment was inspired by an accident. In an unrelated test, a 9,211-byte skill file generated in one shot by a model caused a weaker model to return empty responses six times in a row in the test harness. The team raised the output budget to 16,000 tokens and ran it again—still one out of two came back empty.

The file itself had no bug. It was just too long. The model spent its entire output budget "thinking about the instructions," leaving no allowance for the actual answer. Six consecutive empties wasn't a low-probability event—it was the normal behavior of a naturally bloated file at a 50% failure rate.

To make the conclusion reproducible, the team ran this quantitative experiment: fixed task, fixed system prompt, fixed max_tokens = 16,000, no retries on empty output, with the only variable being skill file size.

Experimental Design: Changing Only One Variable

The task was to extract information from batches of invoices. The skill file embedded three private conventions: prices in cents must be converted to dollars, a 6% channel fee, and one tax-exempt line item listed separately. Scoring used deterministic regex matching for these three conventions, without introducing a judge model.

The approach started from a lean 2,761-byte version and progressively padded it to different target sizes using 54 real redundancy blocks (duplicate sections, restated rules, ritualistic boilerplate that models tend to write themselves), across 7 size points with 4 repetitions each, totaling 28 runs. Before the experiment, they wrote down the falsification condition: if the empty output rate and token consumption did not rise with file size, then the 9.2KB incident was just a coincidence.

The unit of measurement deserves its own note. Bytes are used because that's what you see in the editor; English averages about 4 characters per token, so 6KB is roughly 1,500 tokens of instruction text. The file enters the prompt as input, but it raises the amount of thinking on the output side—and that thinking is exactly what hits the 16,000 limit. The input side isn't free either: the agent loop re-reads the entire conversation every round, so every resident KB is billed repeatedly on every request. The authors' team audited a production pipeline where the workload stayed the same but daily token consumption jumped from 110 million to over 500 million—with the skill directory alone quietly accounting for 161KB in every system prompt.

The relationship between output token limit (max_tokens) and the chain of thought: In models that support reasoning (such as DeepSeek^Ad or Claude's extended thinking), the token stream returned by the model splits into two parts—the visible answer and the invisible thinking process (chain-of-thought). max_tokens limits the sum of both. When the skill file is long, the model needs to "digest" the instructions before generating an answer, and the token consumption of the thinking portion rises significantly; once the chain of thought fills the budget, the model has no room left to output the final answer, and the API returns empty content or truncation. This explains the essence of "empty output" in the experiment: it's not that the model refuses to answer, but that the budget was exhausted during the reasoning phase. File size therefore indirectly affects output success rate by influencing the amount of thinking, rather than directly determining correctness.

Repeated billing in the agent loop: Unlike single-turn Q&A, coding agents typically work in multi-turn loops—each round resends the full conversation history (including the skill file in the system prompt) to the model. This means every KB of the skill file is billed repeatedly on every request, and the cost scales linearly with the number of loop rounds. The "daily token consumption jumping from 110 million to 500 million" mentioned in the experiment is exactly the result of this mechanism: the file itself isn't large, but multiplied by the number of requests per round, the resident overhead becomes astonishing. Trimming skill files therefore not only reduces the risk of single-run failure but also directly cuts ongoing production costs.

What the Data Says

The results across the seven size points fall into two clearly distinct segments.

• 2.8KB: 0% empty output rate, 9,096 average output tokens, 4/4 conventions matched.
• 4.1KB: 0% empty output rate, 7,913 average output tokens, 4/4 conventions matched.
• 6.1KB: 25% empty output rate, 13,598 average output tokens, 3/4 conventions matched.
• 8.0KB: 25% empty output rate, 11,769 average output tokens, 2/4 conventions matched.
• 9.5KB: 25% empty output rate, 15,469 average output tokens, 3/4 conventions matched.
• 11.2KB: 0% empty output rate, 9,927 average output tokens, 4/4 conventions matched.
• Naturally bloated 9.2KB: 50% empty output rate, 14,766 average output tokens, 2/4 conventions matched.

Below 6KB, consumption held steady at 8–9K tokens, far from the 16,000 limit, with zero empty outputs across eight runs. Above 6KB, consumption was pushed into the 12K–15.5K range, with a quarter of the runs dying there. The 9.5KB point averaged 15,469 tokens, almost brushing the limit—two of its four runs survived only because the chain of thought happened to be shorter.

This leads to a counterintuitive detail: whether it crashes depends on the variance in reasoning length, not on anything you can control.

Five Rules More Precise Than "Bigger Is Worse"

Rule one: Below 6KB is the safe zone. The 2.8KB and 4.1KB runs totaled eight executions with zero empty outputs and consumption far from the limit.

Rule two: Above 6KB enters the risk zone. Failure is not a monotonic function of file size but a function of "how close consumption is to the output limit." When you're near the limit, survival is a matter of luck.

Rule three: The 11.2KB point breaks monotonicity. It had zero empty outputs, consumption fell back to 9,927, and it contained all 54 redundancy blocks—including the structured sections that naturally occurring files carry. The team's working hypothesis is that organized content can partially offset sheer volume. This hasn't been verified, and it's kept in to avoid deleting the counterexample and making the conclusion look prettier than the data warrants.

Rule four: Natural bloat is the most toxic kind. At the same size level, the naturally bloated 9.2KB file had a 50% failure rate, while the manually padded 9.5KB file had only 25%. Repetition and verbosity hurt more than byte count. This also explains the original incident in reverse.

Rule five: Volume kills output, not correctness. As long as the model actually produced a result, the convention hit rate was near perfect across all size points. The first question a long file answers isn't "will the model follow it" but "will the model even speak."

Cliff effect and variance dependence: In the experiment, the empty output rate above 6KB did not rise linearly with file size but showed "cliff-like" jumps—once consumption approaches the max_tokens limit, whether the model crashes depends on random fluctuations in chain-of-thought length. Specifically, across different runs of the same file, the number of reasoning steps the model generates may vary slightly; when a given reasoning pass happens to be shorter, the remaining budget suffices to output the answer and the run succeeds; otherwise, the output is empty. This explains why two of the four runs at the 9.5KB size point survived: it's not that the file is "sometimes safe," but that the variance in reasoning length determines which run hits the line. The 6KB threshold is therefore essentially an empirical boundary for "leaving enough headroom for the chain of thought," not a physical limit on file size.

Applying This to Your Own Skill Files

The team currently enforces three rules for the skill files they maintain, AGENTS.md, and long system prompts:

Treat 6KB as a budget ceiling. Not a style suggestion, but an executable range. Their generation pipeline treats 6–9KB as the risk zone (25% failure observed), 9KB and above as high risk, and an audit step blocks files that cross the line.

Delete duplication first, then delete content. The problem with naturally bloated files isn't saying too much—it's saying everything twice. The safety gains from deduplication are greater than those from cutting rules.

The weaker the model, the tighter the budget. The experiment used DeepSeek V4.1-Flash as the executing model, and weak models are precisely where the cliff effect is most pronounced: strong models burn through budgets more comfortably, and the same file can hurt one model without hurting another.

The diagnostic tool is absurdly simple: wc -c is enough.

Boundaries and What Remains Unverified

This measurement has a clear scope: a single task (invoice extraction), a single model family, 4 repetitions per size point, so the 95% confidence interval around "25%" is wide, and algorithmic padding is not entirely equivalent to handwritten bloat. The 11.2KB counterexample still needs a direct test: fix the byte count, vary only the structure, and compare organized padding against verbose padding.

The API cost for the entire gradient experiment was 2.55 RMB, about 0.36 USD. The authors' advice: reproduce first, then draw conclusions. Treat it as a measurement starting point with a clear mechanism, not a universal constant. But the failure mode of "the model returned nothing because it spent its entire budget reading your instructions" is worth checking today against the files you already have.

The underlying tool, Skill Forge, is open-sourced and released in three forms: an MCP server, a prompt pack, and an installable skill.

One-line Summary

28 experiments found that coding agent skill files exceeding 6KB significantly raise the empty output rate, with a naturally bloated 9.2KB file reaching a 50% failure rate.

Paragraph Summary

Based on 28 controlled experiments, the article reveals a practical 6KB upper limit for coding agent skill files. Fixing the task and output budget (16,000 tokens) while varying only file size, it found zero empty outputs below 6KB, roughly 25% empty output above 6KB, and a 50% failure rate for a naturally bloated 9.2KB file. The failure mechanism is that the file inflates output-side thinking, crowding out the answer budget—not that the file itself has a bug. The 11.2KB size point had zero failures, suggesting organized content may partially offset volume, though this remains unverified. Volume primarily kills output rate rather than correctness. The authors recommend treating 6KB as a budget ceiling, prioritizing deduplication, and tightening budgets for weaker models.