BranchOPD
A method that improves online policy distillation (OPD) by performing branch sampling at high-information decision points. It combines shared prefix reuse with protocol boundary repair mechanisms to enhance the efficiency of small models learning from large models.
Timeline (last 90 days)
BranchOPD (Branching On-Policy Distillation) implements branching-sampling distillation through three key steps: informative decision point selection, shared prefix resampling, and token-level teacher supervision.
BranchOPD leverages the student model's own uncertainty and the divergence between the teacher and student to locate the decision points most worth branching.
BranchOPD branches at token-level decisions for math tasks and at within-turn decisions for agent tasks.
Compared to a scheme with seven independent trajectories, shared prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld.
Compared to a scheme with seven independent trajectories, shared prefix reuse reduces student-generated token costs by approximately 31% on WebShop.
BranchOPD introduces a Protocol Boundary Repair (PBR) mechanism to mitigate the problems of repetitive and invalid generation in agent tasks.
On the avg@16 metric for math tasks, BranchOPD improves over Vanilla OPD by 18.55 percentage points.
On the ID/OOD average success rate on ALFWorld, BranchOPD improves over Vanilla OPD by 13.8 percentage points.
BranchOPD outperforms both Vanilla OPD and the more expensive Multi-Rollout OPD (n=7) on all evaluation benchmarks.
Ablation experiments show that branching at uncertain decision points outperforms randomly selecting branching points and fixed turn-initial branching.
1 more timeline events
All Facts (11)
BranchOPD (Branching On-Policy Distillation) implements branching-sampling distillation through three key steps: informative decision point selection, shared prefix resampling, and token-level teacher supervision.
50%UnverifiedBranchOPD leverages the student model's own uncertainty and the divergence between the teacher and student to locate the decision points most worth branching.
50%UnverifiedBranchOPD branches at token-level decisions for math tasks and at within-turn decisions for agent tasks.
50%UnverifiedCompared to a scheme with seven independent trajectories, shared prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld.
50%UnverifiedCompared to a scheme with seven independent trajectories, shared prefix reuse reduces student-generated token costs by approximately 31% on WebShop.
50%UnverifiedBranchOPD introduces a Protocol Boundary Repair (PBR) mechanism to mitigate the problems of repetitive and invalid generation in agent tasks.
50%UnverifiedOn the avg@16 metric for math tasks, BranchOPD improves over Vanilla OPD by 18.55 percentage points.
50%UnverifiedOn the ID/OOD average success rate on ALFWorld, BranchOPD improves over Vanilla OPD by 13.8 percentage points.
50%UnverifiedBranchOPD outperforms both Vanilla OPD and the more expensive Multi-Rollout OPD (n=7) on all evaluation benchmarks.
50%UnverifiedAblation experiments show that branching at uncertain decision points outperforms randomly selecting branching points and fixed turn-initial branching.
50%UnverifiedThe value distribution of supervision is highly uneven, and dynamically allocating supervision resources using the model's own signals is superior to spreading them uniformly.
50%