SAPD
An offline training method for LLM agents that requires no environment interaction. It decomposes multi-turn trajectories into independent decision contexts and combines relative step advantage signals with on-policy distillation, mitigating catastrophic forgetting.
Timeline (last 90 days)
The researchers propose a method called Step-Advantage Policy Distillation (SAPD), with the paper titled "SAPD: Learning from Agent Experience without Environment Interaction".
SAPD uses offline trajectories and relative step-advantage signals to achieve stable policy improvement without new environment observations or multiple rounds of interaction.
SAPD decomposes multi-turn trajectories into independent decision contexts for on-policy distillation.
SAPD was experimented on ALFWorld, WebShop, and search-based question answering tasks.
Experimental results show that SAPD improves performance under different teacher configurations and effectively mitigates catastrophic forgetting compared with SFT.
SAPD runs at least 4.7x faster per training step than online on-policy distillation (OPD).
SAPD's training process requires no new environment observations and no additional multi-turn interaction.
The researchers said they will release SAPD's code soon.
All Facts (8)
The researchers propose a method called Step-Advantage Policy Distillation (SAPD), with the paper titled "SAPD: Learning from Agent Experience without Environment Interaction".
50%UnverifiedSAPD uses offline trajectories and relative step-advantage signals to achieve stable policy improvement without new environment observations or multiple rounds of interaction.
50%UnverifiedSAPD decomposes multi-turn trajectories into independent decision contexts for on-policy distillation.
50%UnverifiedSAPD was experimented on ALFWorld, WebShop, and search-based question answering tasks.
50%UnverifiedExperimental results show that SAPD improves performance under different teacher configurations and effectively mitigates catastrophic forgetting compared with SFT.
50%UnverifiedSAPD runs at least 4.7x faster per training step than online on-policy distillation (OPD).
50%UnverifiedSAPD's training process requires no new environment observations and no additional multi-turn interaction.
50%UnverifiedThe researchers said they will release SAPD's code soon.
50%