SkillScaffold: Teaching Agentic RL to Selectively Inject Skills

When Skill Injection Is No Longer a Given

Enabling large language models (LLMs) to learn complex tasks through multi-turn interaction has become a core idea in agentic reinforcement learning (Agentic RL). Along this path, "skills" — reusable textual guidance distilled from historical interactions — are widely used to improve exploration efficiency. The rationale is natural: if the model can summarize from past successful trajectories "what to do next time in a similar situation," it should theoretically be able to take fewer detours.

But a repeatedly overlooked question is: is skill injection really always beneficial?

Research from SkillScaffold points out that most existing skill-driven RL methods assume by default that injecting skills yields positive returns, or rely on indirect feedback and large numbers of rollouts to evaluate whether a skill is useful. Their pilot experiments reveal a more subtle truth: rollout-based skill utility estimation is not only expensive but also highly unstable under limited sampling; more critically, the same skill can have vastly different effects across different queries and different levels of policy capability — under some queries a skill is like adding wings to a tiger, while under others it becomes interference.

This observation directly points to a design principle: what a skill system needs is not a binary switch of "inject or not," but an adaptive judgment based on the current policy's capability.

Agentic RL differs from traditional RL in that the agent no longer outputs just a single action, but calls tools, retrieves information, and makes step-by-step decisions across multiple turns to ultimately complete long-horizon tasks such as web shopping or multi-step reasoning and manipulation environments. Rewards for such tasks are often sparse and the action space is enormous, making exploration efficiency the main bottleneck. A skill, as textual reusable guidance distilled from historical interaction trajectories, is essentially a kind of "soft prompt" — it does not change the model parameters, but only influences the policy distribution through context, thereby guiding exploration direction without additional gradient updates. This is also why skill-based methods are easier to iterate and transfer than parameter fine-tuning, but precisely because of their conditional effect, determining "when to use one and which one" becomes a key engineering problem.

Core Method: Low-Cost Utility Estimation + Selective Injection

SkillScaffold's solution revolves around a training scaffold composed of three cooperating components.

Reusing Verified Trajectories to Bypass Expensive Rollouts

Traditional methods for evaluating whether a skill is useful often require running environment rollouts specifically for that candidate skill, which is costly and high-variance. SkillScaffold's approach is to reuse training trajectories that have already undergone outcome verification, generate candidate skills on that basis, and estimate skill utility through "outcome-aware likelihood comparison," without relying on candidate-specific environment interaction at all.

This means the marginal cost of utility estimation is greatly reduced, while also avoiding the instability caused by limited sampling.

Outcome-aware likelihood comparison is a utility estimation approach that does not rely on additional environment interaction. The idea is: on a batch of trajectories that have already been run and have verified outcomes (such as success/failure labels), compute the model's log-likelihood of reproducing each trajectory under two conditions — "with the candidate skill" and "without the skill" — and then perform a weighted comparison of the likelihood differences based on the trajectory outcome labels. If the likelihood of successful trajectories rises significantly after injecting a skill, it indicates that the skill is consistent with effective behavior; if the likelihood of failed trajectories is raised by the skill, it suggests that the skill may induce incorrect behavior. Compared with rolling out each candidate skill separately and then counting success rates, this method transforms utility estimation into a single forward computation, with lower variance and lower cost. It is a lightweight borrowing of the classic "off-policy evaluation" idea in RL.

Utility-Driven Selective Injection and Abandonment Mechanism

The estimated utility values directly guide two decisions: whether to inject a skill, and which skill to inject. When the estimated utility of all candidate skills is unsatisfactory, the system chooses skill-free execution rather than forcibly injecting one.

The paper's ablation experiments specifically verify the importance of this "utility-based selection and abandonment" mechanism — it is not an optional engineering detail, but one of the key sources of performance improvement.

Joint Training of Generator and Executor

SkillScaffold includes two roles: the Generator is responsible for producing skills, and the Executor is responsible for executing tasks. The two are jointly trained, enabling the Generator to continuously produce skills adapted to the Executor's current policy level. As the Executor's capability improves, the form and content of the required skills also evolve accordingly.

Ultimately, scaffolded RL training encourages the Executor to internalize skills — being able to complete tasks independently without skills during inference.

Skill internalization refers to the Executor gradually absorbing the behavioral patterns carried by external textual skills into its own policy during training, so that during inference it can complete the same type of task even without provided skills. Its mechanism is similar to teacher-student distillation: early-stage skills serve as strong guidance to help the model find the correct behavior distribution; as the policy is repeatedly reinforced on these behaviors, the model's own prior becomes sufficient to reproduce that behavior, and the marginal utility of the skill declines accordingly — this is precisely why utility estimation can automatically trigger "abandonment." The practical benefit of internalization is reducing inference-time context length and latency, avoiding carrying a huge skill library with every call; its risk is that if there is a shift between the training distribution and the deployment distribution, the internalized capability may fail to generalize, so the quality of the outcome verification signal directly determines the reliability of internalization.

Experimental Performance: Comprehensive Validation on Four Benchmarks

SkillScaffold was evaluated on four benchmarks: ALFWorld, WebShop, ScienceWorld, and τ²-Bench. These environments cover different task types from short-horizon to long-horizon: ALFWorld and ScienceWorld focus on multi-step reasoning and environment manipulation, WebShop examines decision-making in web interaction, and τ²-Bench targets more complex tool use and dialogue scenarios.

The results show that SkillScaffold surpasses strong agentic RL baselines and skill-driven RL baselines on both short-horizon and long-horizon tasks. The paper further supports the necessity of the utility selection and abandonment mechanism through ablations and analysis experiments.

The code is open-sourced on GitHub (https://github.com/Vaidurya00/SkillScaffold).

Why This Direction Is Worth Attention

SkillScaffold touches on a consensus that is taking shape in agentic RL: the value of a skill is conditional. A skill may be necessary guidance when the policy's capability is insufficient, but may become redundant or even interference once the policy has mastered that capability.

This has similarities to the human learning process — beginners need detailed step-by-step prompts, while skilled learners need hands-off practice. SkillScaffold uses quantifiable utility estimation to turn this intuition into a trainable and deployable mechanism.

Another notable design is the goal of skill internalization. Many skill-enhancement methods still need to carry a skill library during inference, bringing additional context overhead and latency. SkillScaffold trains the model to gradually shed its dependence on skills, reducing inference cost while maintaining capability, which has direct significance for real-world deployment.

Of course, the accuracy of utility estimation still depends on the quality and density of the outcome verification signal. Whether this mechanism can remain equally stable on tasks with sparse rewards or difficult verification is a question worth tracking in the future.

One-line Summary

SkillScaffold uses low-cost utility estimation to let agents selectively inject skills, and ultimately internalize skills to break free from dependence.

Paragraph Summary

SkillScaffold challenges the default assumption in agentic reinforcement learning that "skill injection is always beneficial," pointing out that the same skill can vary greatly in utility across different queries and different policy capabilities, and that rollout-based utility estimation is both expensive and unstable. To address this, the method reuses verified training trajectories and achieves low-cost skill utility estimation through outcome-aware likelihood comparison, based on which it decides whether to inject a skill and which one to inject, choosing skill-free execution when all candidate skills have insufficient utility. At the same time, the Generator and Executor are jointly trained, so that skills continuously adapt to the evolving policy capability and push the Executor to internalize skills during inference, completing tasks independently without a skill library. On the four benchmarks ALFWorld, WebShop, ScienceWorld, and τ²-Bench, the method surpasses both agentic RL and skill-driven RL baselines, validating that the utility-driven selection and abandonment mechanism is a key source of performance.