Differentially Private Sparse Reward Estimation: A New Breakthrough in RLHF Privacy Protection

RLHF (Reinforcement Learning from Human Feedback) has become a core technique for aligning large language models, but the preference data it relies on often contains sensitive information. How to efficiently train reward models while protecting user privacy is a key challenge in the current AI safety landscape. A new study proposes a differentially private sparse reward estimation framework, systematically analyzing the privacy-utility trade-off in RLHF under sparsity assumptions for the first time.

The Bottleneck of Existing Methods: The Privacy Cost of Dense Parameterization

Traditional privacy-preserving alignment methods typically assume that reward model parameters are dense, i.e., all dimensions participate in preference modeling. This causes the estimation error upper bound to grow polynomially with the model dimension (d). In today's over-parameterized models with hundreds of billions of parameters, (d) is extremely large, and this dependence causes the cost of privacy protection to rise sharply—either the privacy budget is consumed too quickly, or model utility drops severely.

The study points out that this dense assumption does not match reality: human preferences are often driven by only a few key features. For example, in sentiment generation tasks, the "positive/negative" evaluation may be determined by only a few semantic dimensions. Therefore, leveraging sparsity assumptions can substantially reduce the cost of privacy protection.

In differential privacy, the privacy budget (\varepsilon) is a key parameter measuring the strength of privacy protection: the smaller (\varepsilon) is, the stronger the privacy protection, but the more noise is introduced, leading to higher estimation error. Common privacy definitions include Central Differential Privacy (Central DP) and Local Differential Privacy (Local DP). The former assumes a trusted data curator, while the latter requires users to perturb data locally before uploading. Sample-DP and Label-DP correspond to different granularities of privacy protection, and their privacy budgets are consumed differently, thereby affecting the final utility upper bound.

Sparse RLHF Framework: Theory and Algorithms

The study defines the Private Sparse RLHF problem: assume the reward model is controlled by only (s^* \ll d) relevant features, and preference feedback follows the Bradley-Terry-Luce (BTL) model. Under this setting, the authors establish, for the first time, a complete theoretical framework for differentially private sparse reward estimation under pairwise preference feedback, and design efficient algorithms for two privacy settings:

• Sample-DP: privacy protection applies to the entire sample (i.e., a pair of preference comparisons), requiring that replacing any pair of samples does not significantly affect the output distribution.
• Label-DP: privacy protection applies only to the label (i.e., the preference direction), typically implemented via the randomized response mechanism.

The theoretical analysis provides upper bounds on the squared (\ell_2) parameter estimation error:

• Sample-DP: (\widetilde{O}(s^/n + {s^}^2/(n^2\varepsilon^2)))
• Label-DP (based on randomized response): (\widetilde{O}(s^/n + s^/(n\varepsilon^2)))

where (n) is the amount of training data and (\varepsilon) is the privacy budget. These upper bounds show that the error depends linearly on (s^*), rather than polynomially on (d), and is therefore significantly better in over-parameterized settings.

Lower bound analysis further shows that the Sample-DP rate is near-optimal, and the guarantees of the Label-DP method within the randomized response framework cannot be improved further. The authors also provide a general lower bound for the Label-DP problem, offering a theoretical benchmark for subsequent algorithm design.

The Bradley-Terry-Luce (BTL) model is a probabilistic model commonly used in reward modeling. Its core assumption is that, for a pair of options, the probability that a human prefers one side is related to the exponential ratio of the two reward values, i.e., (P(i \succ j) = \frac{e^{r_i}}{e^{r_i} + e^{r_j}}). This model transforms preference comparisons into constraints on a latent reward function, facilitating theoretical analysis. In RLHF, the BTL model is often used to estimate the reward function from pairwise comparison data; its log-likelihood function has good convexity, but privacy protection introduces additional noise, requiring re-analysis of the estimation error.

Randomized Response is a classic local differential privacy mechanism, originally used for surveys on sensitive questions. Its basic idea is: answer truthfully with a certain probability and give a random answer with another probability, thereby concealing true information at the individual level. In the Label-DP setting, randomized response is used to perturb preference labels, so that an attacker cannot infer a user's true preference from a single label. This mechanism is simple to implement but introduces considerable noise, so its privacy-utility trade-off requires careful theoretical characterization.

The sparsity assumption typically means that the reward function depends on only a few feature dimensions, i.e., the parameter vector is sparse (the number of nonzero elements is (s^)). In statistical learning, sparsity can be exploited through (\ell_1) regularization or greedy algorithms (such as orthogonal matching pursuit), thereby reducing the estimation error from dependence on the total dimension (d) to dependence on the sparsity level (s^). In privacy-preserving settings, sparsity helps reduce the dimensionality of noise injection, because the privacy mechanism only needs to perturb the relevant dimensions, thereby alleviating the curse of dimensionality.

Experimental Validation: A Better Privacy-Utility Trade-off

On sentiment generation and safety alignment benchmarks, the framework shows significant advantages over dense baselines. Experimental results indicate that the sparse method achieves lower estimation error under the same privacy budget, thereby achieving a better trade-off between model alignment quality and privacy protection. This is particularly important for practical deployment scenarios that need to handle large amounts of user preference data—meeting compliance requirements while maintaining model performance.

Implications for RLHF Privacy Protection

The core contribution of this work is introducing sparsity into the theoretical analysis of private RLHF, breaking the constraint of the curse of dimensionality on privacy protection. For practitioners, this means that when designing privacy-preserving alignment solutions, sparse reward parameterization can be prioritized rather than blindly increasing the privacy budget. At the same time, the rate difference between Label-DP and Sample-DP also suggests that if the application scenario allows privacy protection only on labels (e.g., collecting preferences via randomized response), better utility can be achieved.

However, the study is still based on the BTL model and linear reward assumptions, and future work needs to extend to more complex preference models (such as deep neural network reward models) and broader privacy definitions (such as combinations of local differential privacy and central differential privacy). Nevertheless, its theoretical framework lays an important foundation for the sparse design of privacy-preserving RLHF.

One-line Summary

A new study proposes a differentially private sparse reward estimation framework that achieves a better privacy-utility trade-off in RLHF.

Paragraph Summary

This study addresses the privacy protection of preference data in RLHF by proposing a differentially private sparse reward estimation framework. Traditional methods assume dense reward model parameters, causing the cost of privacy protection to grow polynomially with model dimension. The new framework assumes that rewards are controlled by only a few key features and, based on the BTL model, designs efficient algorithms for two privacy settings: Sample-DP and Label-DP. Theoretical analysis shows that the estimation error upper bound depends linearly on sparsity, avoiding the curse of dimensionality; lower bound analysis proves that the Sample-DP rate is near-optimal. Experiments on sentiment generation and safety alignment tasks verify that the sparse method has lower estimation error and a better privacy-utility trade-off under the same privacy budget.