SeTPO
A policy optimization method for tool learning in small language models. It combines bidirectional KL constraints with student autonomous exploration, selective teacher guidance, and continuous demonstration supervision. On low-reward trajectories it provides token-level teacher feedback via reverse KL, while globally retaining forward KL supervision to prevent forgetting.
Timeline (last 90 days)
SeTPO stands for Selective Teacher-guided Policy Optimization, a method for tool learning in small language models.
SeTPO introduces teacher signals only for low-reward trajectories, uses reverse KL divergence to derive token-level teacher feedback, and incorporates it into the policy advantage function.
SeTPO retains an auxiliary forward KL supervision objective throughout training, to prevent behaviors demonstrated by the teacher from being forgotten or degrading during training.
The researchers provide a theoretical analysis under an idealized bidirectional KL objective, proving that the expected reward of the optimal policy is no lower than the teacher's, while retaining a guaranteed proportion of the teacher's probability mass.
On the two tool-learning benchmarks BFCL v4 and API Bank, SeTPO was compared against three classes of baselines: supervised fine-tuning, reinforcement learning, and distillation.
At all three student model scales, SeTPO achieves better overall performance than the baseline methods.
The core design idea of SeTPO is: the relationship between teacher guidance and student exploration is not necessarily zero-sum, and selective intervention can provide precise help when the student needs it without suppressing its exploration ability.
All Facts (7)
SeTPO stands for Selective Teacher-guided Policy Optimization, a method for tool learning in small language models.
50%UnverifiedSeTPO introduces teacher signals only for low-reward trajectories, uses reverse KL divergence to derive token-level teacher feedback, and incorporates it into the policy advantage function.
50%UnverifiedSeTPO retains an auxiliary forward KL supervision objective throughout training, to prevent behaviors demonstrated by the teacher from being forgotten or degrading during training.
50%UnverifiedThe researchers provide a theoretical analysis under an idealized bidirectional KL objective, proving that the expected reward of the optimal policy is no lower than the teacher's, while retaining a guaranteed proportion of the teacher's probability mass.
50%UnverifiedOn the two tool-learning benchmarks BFCL v4 and API Bank, SeTPO was compared against three classes of baselines: supervised fine-tuning, reinforcement learning, and distillation.
50%UnverifiedAt all three student model scales, SeTPO achieves better overall performance than the baseline methods.
50%UnverifiedThe core design idea of SeTPO is: the relationship between teacher guidance and student exploration is not necessarily zero-sum, and selective intervention can provide precise help when the student needs it without suppressing its exploration ability.
50%