SGUID: Distilling a Lean Skill Library for Co-Evolution of LLM Skills and Models

Large language models can significantly improve downstream task performance by injecting "skills"—reusable procedural guidance—at inference time. But a long-overlooked question is: not all skills are worth distilling. A recent study, SGUID, points out that in on-policy distillation, fewer than 25% of retrieved skills provide effective learning signals.

The Hidden Waste in Skill Distillation

Previous methods typically rely on semantic relevance to retrieve skills from a skill library, then use them as inference-time patches or for model distillation. This "on-demand invocation" approach seems reasonable, but it masks a key fact: the utility of individual skills varies enormously.

Under an on-policy distillation setting, the research team had a skill-conditioned policy act as the teacher model. They found that fewer than one-quarter of retrieved skills actually provided useful distillation signals. In other words, most content in traditional skill libraries is just "along for the ride" during distillation—and may even introduce noise.

This finding points directly to a methodological issue: rather than feeding the entire skill library into the model, it's better to first filter out the subset of skills that truly have value.

On-policy distillation is a variant of knowledge distillation, distinct from traditional offline distillation. In traditional distillation, the teacher model provides soft labels on a fixed batch of samples—usually generated by other models—and the student model passively fits those labels. In on-policy distillation, the student model first generates its own rollouts, and then the teacher model scores or provides supervision on these "paths the student itself has taken." The advantage is that the training distribution matches the student's actual inference distribution, avoiding distribution shift between offline data and student behavior. In the skill distillation setting, the teacher model is the same model "conditioned on a certain skill"—that is, the skill is injected as a prompt so it acts as a teacher that better understands that skill, thereby distilling this procedural knowledge back into the student model without the skill prompt.

How SGUID Filters Skills

The core logic of SGUID is simple yet strict: a skill is retained only if it continuously produces effective learning signals during training. This criterion of "continuous effectiveness" filters out skills that are occasionally useful but whose overall contribution is unstable.

After filtering, the selected skills are distilled into the model, producing a stronger student model. This process is not one-off—SGUID is designed to support "model-skill co-evolution."

Specifically, after one round of distillation, the system organizes a new candidate skill library from the rollouts of the updated model, and SGUID then selects the skills to be internalized in the next round. This cycle repeats, with the model and skill library driving each other and iterating together.

Model-skill co-evolution draws on ideas from continual learning and self-play-style iteration. Traditional skill libraries are static: once skills are collected, they remain fixed, and the model can only passively learn from the same batch of skills repeatedly, making diminishing returns likely. Co-evolution breaks this static assumption by letting the skill library update as the model's capabilities change—as the model becomes stronger, new behavioral patterns emerge in rollouts, these patterns are distilled into new candidate skills, which in turn drive the next round of model evolution. This creates a positive feedback loop, but also introduces risk: if a poor-quality skill is mistakenly included in one round, the error can be amplified and contaminate all subsequent rounds. Therefore, the filtering mechanism is even more critical in co-evolution than in static distillation, and SGUID's continuous-effectiveness criterion provides a stable anchor for this loop.

Experimental Results: 6 Skills Match the Full Skill Library

Across four models from the Olmo and Qwen families, SGUID demonstrates surprising efficiency. With just 6 selected skills distilled, on three of the four models, the average avg@12 already matches or exceeds the effect of distilling the full skill library. After a second round of distilling 3 newly selected skills, all four models reach or exceed the full-library level—while the full skill library is at most 11 times larger than the skills selected by SGUID.

The stability of the co-evolution loop is validated on Qwen3-8B: after selecting 3 new skills in the second round, model performance improves from 64.3% to 66.3%.

avg@12 is a common aggregate metric for inference evaluation. It means sampling the model 12 times for the same set of questions and taking the average accuracy across questions. Compared with single-pass greedy decoding (pass@1), avg@12 reduces the variance caused by sampling randomness and more stably reflects the model's true capability distribution on difficult problems. On math and reasoning benchmarks (such as competition problems like HMMT25), because single-sample fluctuations are large, avg@12 is widely used. Understanding this metric helps interpret numbers like "from 64.3% to 66.3%" in the text: it represents the change in average accuracy under multiple samples, and an improvement of a few percentage points is already significant on competition-level tasks, since such problems are inherently difficult and baseline accuracy is limited.

The Selection Mechanism Is Key to Stability

The most cautionary finding in the study comes from a comparison experiment on Qwen3-4B. If SGUID filtering is not used and the model is directly updated with unfiltered skills, performance actually declines—dropping by 0.3 percentage points on HMMT25.

SGUID's performance is the exact opposite: after the first round, HMMT25 improves by 0.5 percentage points, and after the second round, cumulative improvement reaches 1.1 percentage points. This positive-versus-negative contrast shows that skill filtering is not an optional optimization, but a necessary mechanism for stable co-evolution. Without filtering, the model may "go off track" on erroneous skill signals; with filtering, each round of iteration accumulates positive gains.

This research elevates "which skills to distill" from an overlooked detail to a core issue that determines the success or failure of model-skill co-evolution. For researchers and engineers building skill libraries or exploring continual learning for models, SGUID offers a filtering paradigm worth learning from.