Is Majority-Vote Pseudo-Labeling Unreliable? PC-Dr.DPO Makes TTRL More Robust

Test-Time Reinforcement Learning (TTRL) enables pretrained language models to adapt to unlabeled target-domain data using self-generated pseudo-labels. However, the reliability of these pseudo-labels has long lacked in-depth understanding. A recent paper systematically analyzes the majority-vote supervision mechanism commonly used in TTRL and proposes an improved method.

Error Decomposition of Majority-Vote Supervision

The paper decomposes the majority-vote supervision error under online Direct Preference Optimization (DPO) into two components: population-majority bias and prompt-dependent finite-rollout variation.

Population-majority bias stems from the model's preference for the population-level majority answer, a bias that is typically unidentifiable and irremovable without external supervision. The authors therefore focus on mitigating the downstream impact of finite-rollout variation. Finite-rollout variation refers to the instability of voting results caused by sampling only a limited number of responses per prompt, which in turn affects pseudo-label quality.

This decomposition reveals the core reason for unreliable pseudo-labels in TTRL: even though majority voting can theoretically reduce noise, in practice the limited number of samples and differences across prompts still introduce significant variation.

Direct Preference Optimization (DPO) is a reinforcement learning fine-tuning method that requires no explicit reward model. It directly optimizes the policy by maximizing the likelihood ratio of preference pairs (chosen/rejected), replacing the PPO pipeline in traditional RLHF. In the TTRL setting, the model samples multiple responses for the same prompt, then uses majority voting to decide "pseudo preference pairs," thereby constructing DPO training signals without human annotation.

Population-majority bias draws on the "population-level" concept from statistical estimation: if the model has a systematic preference for a certain type of question (e.g., always tending to output a certain format or common answer), then even as the number of samples approaches infinity, majority voting will converge to that preference rather than the true correct answer. This bias is prompt-independent and constitutes a structural error arising from the model's prior or training data distribution.

Finite-rollout variation corresponds to sampling variance in statistics: each prompt samples only a limited number of responses (e.g., 8 or 16), so voting results fluctuate randomly with sampling. This fluctuation is especially severe for prompts where the model itself has high uncertainty, causing pseudo-labels to jump between adjacent training steps and introducing noise into optimization.

PC-Dr.DPO: Prompt-Conditional Distributionally Robust Optimization

To address finite-rollout variation, the authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO). This method performs robust reweighting within each prompt based on observed voting reliability, making updates from sampling-sensitive prompts more conservative.

Specifically, PC-Dr.DPO assigns different weights to each prompt's voting results during optimization, lowering the weight for prompts with low voting reliability and thereby reducing the impact of noisy pseudo-labels on model updates. This conditional robustness design makes the model more stable when adapting to the target domain.

In addition, the authors introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. This mechanism preserves previously learned knowledge by caching historical information, preventing the model from losing its original capabilities while adapting to new data.

Distributionally Robust Optimization (DRO) is an optimization framework that minimizes risk under the worst-case distribution, commonly used to handle mismatches between training and test distributions. Standard DRO finds the worst distribution within an uncertainty set and optimizes its loss, thereby achieving robustness to distribution shift. PC-Dr.DPO conditions this idea on each prompt: based on an estimate of that prompt's voting reliability, it constructs a local uncertainty set around the empirical voting distribution and optimizes the DPO loss under the worst case within the set. The lower the reliability, the larger the uncertainty set, which is equivalent to imposing stronger regularization or downweighting on updates for that prompt.

Self-supervised forgetting refers to the gradual loss of general capabilities acquired during pretraining when a model performs online adaptation using only self-generated labels. Without ground-truth labels as anchors, the model may overfit to the noise in the current batch of pseudo-labels, leading to performance degradation on the original distribution. The cache mechanism mitigates this catastrophic forgetting by storing historical responses or historical policy outputs and adding consistency constraints during updates.

Experimental Validation and Results

The paper conducts experiments on six math- and knowledge-intensive reasoning benchmarks using five model backbones. Results show that across 30 settings, PC-Dr.DPO matches or exceeds the strongest TTRL baseline in 24 settings.

This result indicates that explicitly handling finite-rollout variation yields consistent performance gains, especially on sampling-sensitive tasks. Although population-majority bias cannot be fully eliminated, PC-Dr.DPO significantly improves TTRL reliability by controlling the downstream impact of variation.

Implications and Outlook

This study reveals a key issue in pseudo-label reliability within TTRL and provides a practical solution. For researchers working on test-time adaptation and reinforcement learning, the robust reweighting idea of PC-Dr.DPO can be extended to other self-supervised scenarios. Future work may explore how to incorporate external supervision signals to further reduce population-majority bias, and how to combine the cache mechanism with more complex online learning algorithms.

Overall, this research reminds us that majority voting is not a panacea—under finite sampling, optimization objectives still need to be designed carefully. PC-Dr.DPO offers a new direction for more reliable test-time reinforcement learning.

One-line Summary

This paper decomposes the error of majority voting in TTRL and proposes PC-Dr.DPO, which reweights by prompt reliability to improve pseudo-label reliability.

Paragraph Summary

This paper systematically analyzes the reliability of majority-vote pseudo-labels in Test-Time Reinforcement Learning (TTRL), decomposing supervision error into an irremovable population-majority bias and a mitigable prompt-dependent finite-rollout variation. To address the latter, the authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which performs robust reweighting within each prompt based on voting reliability, making updates from sampling-sensitive prompts more conservative, and introduces a cache mechanism to reduce self-supervised forgetting during online adaptation. Across 30 settings on six math and knowledge reasoning benchmarks with five model backbones, PC-Dr.DPO matches or surpasses the strongest TTRL baseline in 24 settings, demonstrating that explicitly handling finite-rollout variation can significantly improve TTRL reliability.