Finite Rollout Variance
Instability in voting outcomes caused by sampling only a limited number of responses per prompt, corresponding to sampling variance in statistics.
Timeline (last 90 days)
The paper decomposes the majority-vote supervision error under online direct preference optimization (DPO) into two parts: population-majority bias and prompt-dependent finite-rollout variation.
Finite-rollout variation refers to the instability of voting results caused by sampling only a limited number of responses per prompt, which in turn affects the quality of pseudo-labels.
Explicitly handling finite-rollout variation yields consistent performance improvements, especially on sampling-sensitive tasks.
All Facts (3)
The paper decomposes the majority-vote supervision error under online direct preference optimization (DPO) into two parts: population-majority bias and prompt-dependent finite-rollout variation.
50%UnverifiedFinite-rollout variation refers to the instability of voting results caused by sampling only a limited number of responses per prompt, which in turn affects the quality of pseudo-labels.
50%UnverifiedExplicitly handling finite-rollout variation yields consistent performance improvements, especially on sampling-sensitive tasks.
50%