[KongchangAI]
Model提示条件分布鲁棒 DPO / Prompt-Conditional Distributionally Robust DPO

PC-Dr.DPO

An improved method for test-time reinforcement learning that performs robust reweighting within prompts according to voting reliability, reducing the update weight of low-reliability prompts to mitigate the effects of finite rollout variance.

Timeline (last 90 days)

Nov 9

The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which performs robust reweighting within prompts based on the observed voting reliability, making updates from sampling-sensitive prompts more conservative.

Unverified50%
Nov 9

During optimization, PC-Dr.DPO assigns different weights to the voting results of each prompt, lowering the weights of prompts with low voting reliability and thereby reducing the impact of noisy pseudo-labels on model updates.

Unverified50%
Nov 9

PC-Dr.DPO introduces a cache-based mechanism to reduce self-supervised forgetting during online adaptation.

Unverified50%
Nov 9

The paper conducts experiments on six math and knowledge-intensive reasoning benchmarks, using five model backbones.

Unverified50%
Nov 9

Among 30 settings, PC-Dr.DPO matches or exceeds the strongest TTRL baseline in 24 settings.

Unverified50%
Nov 9

Explicitly handling finite-rollout variation yields consistent performance improvements, especially on sampling-sensitive tasks.

Unverified50%
Nov 9

Although population-majority bias cannot be fully eliminated, PC-Dr.DPO significantly improves the reliability of TTRL by controlling the downstream effects of variation.

Unverified50%

All Facts (7)

Source Articles