PC-Dr.DPO
An improved method for test-time reinforcement learning that performs robust reweighting within prompts according to voting reliability, reducing the update weight of low-reliability prompts to mitigate the effects of finite rollout variance.
Timeline (last 90 days)
The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which performs robust reweighting within prompts based on the observed voting reliability, making updates from sampling-sensitive prompts more conservative.
During optimization, PC-Dr.DPO assigns different weights to the voting results of each prompt, lowering the weights of prompts with low voting reliability and thereby reducing the impact of noisy pseudo-labels on model updates.
PC-Dr.DPO introduces a cache-based mechanism to reduce self-supervised forgetting during online adaptation.
The paper conducts experiments on six math and knowledge-intensive reasoning benchmarks, using five model backbones.
Among 30 settings, PC-Dr.DPO matches or exceeds the strongest TTRL baseline in 24 settings.
Explicitly handling finite-rollout variation yields consistent performance improvements, especially on sampling-sensitive tasks.
Although population-majority bias cannot be fully eliminated, PC-Dr.DPO significantly improves the reliability of TTRL by controlling the downstream effects of variation.
All Facts (7)
The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which performs robust reweighting within prompts based on the observed voting reliability, making updates from sampling-sensitive prompts more conservative.
50%UnverifiedDuring optimization, PC-Dr.DPO assigns different weights to the voting results of each prompt, lowering the weights of prompts with low voting reliability and thereby reducing the impact of noisy pseudo-labels on model updates.
50%UnverifiedPC-Dr.DPO introduces a cache-based mechanism to reduce self-supervised forgetting during online adaptation.
50%UnverifiedThe paper conducts experiments on six math and knowledge-intensive reasoning benchmarks, using five model backbones.
50%UnverifiedAmong 30 settings, PC-Dr.DPO matches or exceeds the strongest TTRL baseline in 24 settings.
50%UnverifiedExplicitly handling finite-rollout variation yields consistent performance improvements, especially on sampling-sensitive tasks.
50%UnverifiedAlthough population-majority bias cannot be fully eliminated, PC-Dr.DPO significantly improves the reliability of TTRL by controlling the downstream effects of variation.
50%