Reward-Free Filtering
A data filtering method that, in the absence of environment reward signals, filters low-quality teacher takeover trajectories through a multi-sample consensus mechanism or LLM judging, preserving the reward-free property.
Timeline (last 90 days)
The R2S pipeline has three steps: 1) starting from the untrained student model, mine the states where it diverges most from the teacher; 2) run teacher takeovers from these states and filter the takeover segments using reward-free checks (consensus or LLM judging); 3) fine-tune the teacher scorer on the filtered takeover data, then freeze it as the sole scorer for standard OPD.
The reward-free filtering uses a consensus mechanism (sampling takeovers multiple times and keeping segments with consistent outcomes) or LLM judging (a large model judges whether the takeover is reasonable) to filter out low-quality trajectories, and the entire process does not rely on task success signals.
All Facts (2)
The R2S pipeline has three steps: 1) starting from the untrained student model, mine the states where it diverges most from the teacher; 2) run teacher takeovers from these states and filter the takeover segments using reward-free checks (consensus or LLM judging); 3) fine-tune the teacher scorer on the filtered takeover data, then freeze it as the sole scorer for standard OPD.
50%UnverifiedThe reward-free filtering uses a consensus mechanism (sampling takeovers multiple times and keeping segments with consistent outcomes) or LLM judging (a large model judges whether the takeover is reasonable) to filter out low-quality trajectories, and the entire process does not rely on task success signals.
50%