CleanRM
A paper on training reward models by calibrating noisy labels with clean preference data. It aims to obtain quality approaching that of clean annotation from inexpensive and noisy labels.
Timeline (last 90 days)
Training a reward model directly on noisy preference data yields biased risk estimates.
The core method of CleanRM is to calibrate the noise ratio on a per-sample basis rather than estimating a global noise rate.
CleanRM uses Optimal Transport to align the noisy preference distribution with a small clean feedback distribution.
CleanRM constructs a corrected surrogate loss function to train the reward model, and theoretically proves that this objective function is unbiased with respect to the clean risk.
The entire CleanRM pipeline requires only a small amount of clean feedback to denoise the whole preference dataset.
CleanRM surpasses current state-of-the-art baseline methods on denoising metrics.
CleanRM demonstrates more effective alignment results in the more critical downstream RLHF alignment task.
The transport plan matrix of optimal transport provides per-sample-level structural information, which is the key technical prerequisite for achieving per-sample calibration rather than global uniform denoising.
The computational overhead of optimal transport, the sampling strategy for clean feedback, and the stability of noise ratio estimation under extremely sparse clean data are issues that need to be weighed in deployment.
All Facts (9)
Training a reward model directly on noisy preference data yields biased risk estimates.
50%UnverifiedThe core method of CleanRM is to calibrate the noise ratio on a per-sample basis rather than estimating a global noise rate.
50%UnverifiedCleanRM uses Optimal Transport to align the noisy preference distribution with a small clean feedback distribution.
50%UnverifiedCleanRM constructs a corrected surrogate loss function to train the reward model, and theoretically proves that this objective function is unbiased with respect to the clean risk.
50%UnverifiedThe entire CleanRM pipeline requires only a small amount of clean feedback to denoise the whole preference dataset.
50%UnverifiedCleanRM surpasses current state-of-the-art baseline methods on denoising metrics.
50%UnverifiedCleanRM demonstrates more effective alignment results in the more critical downstream RLHF alignment task.
50%UnverifiedThe transport plan matrix of optimal transport provides per-sample-level structural information, which is the key technical prerequisite for achieving per-sample calibration rather than global uniform denoising.
50%UnverifiedThe computational overhead of optimal transport, the sampling strategy for clean feedback, and the stability of noise ratio estimation under extremely sparse clean data are issues that need to be weighed in deployment.
50%