SAPD: Efficiently Training LLM Agents from Offline Trajectories Without Environment Interaction

Introduction: The Bottleneck of LLM Agent Training

Training large language model (LLM) agents typically requires a large number of executable environments, which are costly to build, maintain, and run, severely constraining the scaling of training experience. Although existing offline trajectories can reuse interaction experience, effectively optimizing the current policy without revisiting the environment remains a major challenge: supervised fine-tuning (SFT) can lead to catastrophic forgetting, while online reinforcement learning requires real-time interaction with the environment.

To address this problem, researchers have proposed a new method—Step-Advantage Policy Distillation (SAPD)—presented in the paper titled "SAPD: Learning from Agent Experience without Environment Interaction." This method leverages offline trajectories and relative step-advantage signals to achieve stable policy improvement without new environment observations or multi-turn interactions.

Catastrophic forgetting refers to the phenomenon in which a model rapidly loses previously acquired knowledge when learning a new task or data distribution. In LLM agent training, if supervised fine-tuning is performed directly on offline trajectories, the model overfits to the action distribution in those trajectories, deviating from its original general capabilities and policy priors, which leads to degraded performance on unseen environments or tasks. This issue is one of the main risks of the offline training paradigm and is the direct motivation for SAPD's emphasis on distillation rather than pure SFT for stable optimization.

On-policy reinforcement learning requires the agent to continuously interact with the environment, collect new trajectories, and update the policy during training. Its advantage is that the data distribution matches the current policy, but the cost is that the environment must be executable and scalable, and the interaction cost grows linearly with training scale. For LLM agent training that requires large numbers of parallel environments, this cost is often prohibitive, so researchers have turned to using existing offline trajectories to approximate the effect of on-policy optimization.

SAPD Core Idea: Decomposing Trajectories and Advantage Signals

The core innovation of SAPD lies in decomposing multi-turn trajectories into independent decision contexts for on-policy distillation. Specifically, the student model generates responses based on recorded history, and relative step-advantage signals are introduced to stabilize training. The entire training process requires no new environment observations and no additional multi-turn interactions.

The Role of Relative Step Advantage

The relative step-advantage signal is a key component of SAPD. By comparing the decision quality of different steps, it provides more fine-grained guidance to the student model, thereby avoiding the bias accumulation that may occur in traditional distillation. This design enables SAPD to efficiently extract knowledge from offline data while maintaining training stability.

On-policy distillation refers to a setup in which the student model samples from its own generated distribution, and the teacher model then provides supervision signals for these samples. Compared with traditional offline distillation, it narrows the distribution gap between training and inference, but the prerequisite is that the student can generate reasonable decision contexts. SAPD splits multi-turn trajectories into independent single-step decision contexts, allowing the student to generate responses under recorded historical conditions, thereby simulating the on-policy distillation process without any contact with the environment.

Step-level advantage signals draw on the idea of the advantage function in reinforcement learning: advantage measures how good an action is relative to a baseline policy, while step-level advantage refines this comparison to each decision point in a trajectory. Compared with using only the final return or a scalar reward for the entire trajectory, step-level signals can distinguish which steps in the same trajectory are more critical and which contribute less, thereby alleviating the credit assignment problem and reducing bias accumulation during distillation.

Experimental Validation: Performance Gains Across Multiple Environments

The researchers conducted experiments on ALFWorld, WebShop, and search-based question answering tasks. Results show that SAPD improves performance under different teacher configurations, effectively mitigates catastrophic forgetting compared with SFT, and runs at least 4.7 times faster per training step than online on-policy distillation (OPD).

Dual Advantages in Performance and Efficiency

SAPD demonstrates stable performance improvements across multiple benchmarks, while training efficiency is significantly better than online methods. This makes it a promising approach for training LLM agents under limited computational resources and environmental constraints.

Conclusion and Outlook

SAPD provides an offline training paradigm that requires no environment interaction, opening a new path for large-scale training of LLM agents. The researchers say code will be released soon, which is expected to further promote the application and development of this field.

For AI researchers and developers, the significance of SAPD lies in lowering the environmental dependency threshold for agent training, making it possible to leverage existing offline data. In the future, this method may be extended to more complex tasks and larger-scale model training.