CleanRM: Cracking the RLHF Noisy Reward Modeling Challenge with a Few Clean Feedback

In the RLHF (Reinforcement Learning from Human Feedback) pipeline, the quality of the reward model directly determines the effectiveness of large model alignment. However, real-world preference data is almost never completely clean—the ambiguity of annotators' subjective judgments, oversights in manual annotation, and even the systematic biases introduced when using LLMs as judges all cause feedback signals to carry noise.

A research paper titled "CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF" points out that directly training a reward model on noisy preference data yields biased risk estimates. While classical denoising methods can be applied, they generally assume that the noise ratio is uniform across samples, whereas noise in preference data is precisely sample-dependent. CleanRM attempts to solve this structural challenge.

Why Preference Noise Cannot Be Solved by "Unified Denoising"

Traditional denoising learning typically assumes that the probability of each sample being mislabeled is the same constant. Under this assumption, mature techniques such as loss correction and sample reweighting can be used to approximate the clean distribution.

But the noise patterns in preference data are completely different. A judgment that "A is better than B" may flip because the annotator was in a different state that day, may be a random guess because the question domain is too specialized, or may be systematically skewed because the LLM judge has a preference for certain types of phrasing. This means that the noise ratio varies greatly across samples, and modeling with a single global parameter only introduces new bias.

The core insight of CleanRM is: rather than estimating a global noise rate, it is better to calibrate the noise ratio on a per-sample basis. And the anchor for this calibration is a small amount of manually verified clean feedback.

RLHF (Reinforcement Learning from Human Feedback) is the mainstream paradigm for large language model alignment. Its typical pipeline consists of three steps: first, the model generates multiple candidate responses to the same prompt, and human annotators (or LLM judges) compare these responses pairwise and provide preference rankings, forming a preference dataset; then a reward model is trained on this preference data to learn to assign higher scores to more preferred responses; finally, the reward model's scores serve as the optimization objective, and reinforcement learning algorithms (such as PPO) are used to fine-tune the policy model. The reward model plays the role of a "proxy for human preferences," so its quality directly determines the direction in which the policy model will ultimately drift. If the preference signals used to train the reward model are themselves noisy, then the reward model learns a distorted preference function, and the policy model will over-optimize along this distorted signal—the so-called "reward hacking"—ultimately producing responses that appear to score highly but do not actually align with true human preferences. This is precisely the fundamental reason why CleanRM emphasizes the importance of denoising in reward modeling.

CleanRM's Method: Per-Sample Calibration via Optimal Transport

The technical approach of CleanRM can be broken down into three components.

First, distribution alignment. It aligns the joint preference distribution "recovered" from noisy feedback with the distribution formed by a small amount of clean feedback. The tool used here is Optimal Transport (OT), which finds the correspondence between noisy samples and clean samples by minimizing the transport cost between the two distributions.

Second, per-sample noise ratio estimation. During the distribution alignment process, each noisy sample is assigned a sample-level noise ratio, rather than sharing the same parameter. This allows the model to distinguish between "this annotation is highly reliable" and "this annotation is essentially random."

Third, noise-corrected surrogate loss. After obtaining per-sample noise ratios, CleanRM constructs a corrected surrogate loss function to train the reward model. The key point is that the authors theoretically prove that this objective function is unbiased with respect to the clean risk—this is the core guarantee that distinguishes it from heuristic denoising methods.

It is worth noting that the entire pipeline requires only a "small amount" of clean feedback. This means that in practical deployment, the annotation team only needs to sample and carefully label a small portion of the data to leverage denoising effects across the entire preference dataset.

Optimal Transport (OT) originates from the transportation problem proposed by Monge and Kantorovich: given two probability distributions, how to move one distribution to the other with minimum total transport cost while satisfying mass conservation constraints. Its continuous form's optimal value is the Wasserstein distance, while the discrete form reduces to a linear programming problem that can be efficiently approximated via the Sinkhorn algorithm with entropy regularization. In machine learning, OT is often used as a tool for alignment and matching between distributions, because it not only provides a numerical distance but also returns a "transport plan" matrix that explicitly characterizes the correspondence of how each sample in the source distribution is mapped to samples in the target distribution. CleanRM leverages this property: treating the noisy preference distribution as the source distribution and the small clean feedback distribution as the target distribution, it solves the OT to obtain correspondence weights between samples, and then infers individual noise ratios for each noisy sample. Compared to metrics that only output a single distance, OT's transport plan provides sample-level structural information—this is the key technical prerequisite that enables per-sample calibration rather than global unified denoising.

Experimental Results and Downstream Alignment Benefits

The paper reports extensive experimental comparisons. CleanRM surpasses state-of-the-art baseline methods on denoising metrics, and also demonstrates more effective alignment results in the more critical downstream RLHF alignment tasks.

The significance of this result lies in the fact that denoising the reward model is not just about "making the loss numbers look good"—it genuinely propagates to the final policy model's behavior. A less noisy reward signal guides policy optimization toward directions that better align with true human preferences.

For teams building RLHF pipelines, the practical value of this work lies in lowering the threshold for clean data requirements. Fully fine-labeling preference data is extremely costly, and CleanRM's approach is: use a small amount of clean feedback as a "measuring stick," characterize the noise structure through optimal transport, and then correct the entire dataset in reverse.

Unbiased risk estimation is a core concept in statistics and denoising learning. Let the true risk on clean data be R_clean. If the surrogate loss L_noisy constructed from noisy data equals R_clean in expectation, i.e., E[L_noisy] = R_clean, then the surrogate loss is said to be unbiased with respect to the clean risk. Unbiasedness is important because it guarantees that when the data volume is sufficiently large, minimizing the noisy objective is equivalent to minimizing the true objective, and the model will not systematically converge in the wrong direction. Many heuristic denoising methods (such as hard threshold filtering and fixed-discount reweighting) are only empirically effective and lack unbiased guarantees, potentially introducing new systematic bias when the noise structure is complex. CleanRM theoretically proves that its noise-corrected surrogate loss is unbiased with respect to the clean risk, meaning it provides not a "looks better" heuristic patch, but a consistent estimator with statistical guarantees. This is also the theoretical support for why it can simultaneously surpass existing baselines on both denoising metrics and downstream RLHF alignment performance.

Implications for RLHF Engineering Practice

CleanRM reveals an easily overlooked fact: noise in preference data is not uniform, and therefore cannot be handled with a one-size-fits-all approach.

From an engineering perspective, this suggests that when designing annotation workflows, teams should focus not only on the total volume of annotations but also on acquiring "clean anchors"—even if the proportion is small, as long as they cover enough preference patterns, they can support per-sample noise modeling. At the same time, evaluating denoising objectives directly in connection with downstream alignment performance is more meaningful than simply looking at the reward model's classification accuracy.

Of course, the computational overhead of optimal transport, the sampling strategy for clean feedback, and the stability of noise ratio estimation under extremely sparse clean data remain issues to be weighed in practical deployment. But the overall direction provides a new approach with theoretical guarantees for noise-robust reward modeling.

One-line Summary

CleanRM uses a small amount of clean feedback and optimal transport to calibrate preference noise on a per-sample basis, achieving unbiased reward modeling.

Paragraph Summary

To address the problem that preference data in RLHF is generally noisy and classical denoising methods fail due to their assumption of uniform noise, CleanRM proposes a novel approach of per-sample noise ratio calibration. It uses a small amount of manually fine-labeled clean feedback as anchors, aligns the noisy preference distribution with the clean distribution through optimal transport, estimates a sample-level noise ratio for each sample, and constructs a noise-corrected surrogate loss to train the reward model. The authors theoretically prove that this objective is unbiased with respect to the clean risk, avoiding the new biases introduced by heuristic denoising. Experiments show that CleanRM surpasses existing state-of-the-art baselines on denoising metrics and delivers more effective downstream RLHF alignment, reducing dependence on fully fine-labeled data.