[KongchangAI]
· 1 min read· 655 words

FEROM: A New Marginal Policy Optimization Method for Masked Diffusion Language Models

FEROM: A New Marginal Policy Optimization Method for Masked Diffusion Language Models

FEROM proposes Rao-Blackwellized policy gradients and frontier reveal marginalization to reduce optimization variance for masked diffusion language models.

This paper addresses bias and high variance in policy optimization for masked diffusion language models caused by reveal order as an endogenous latent variable, proposing FEROM. It derives a Rao-Blackwellized policy gradient identity expressed as posterior edge occupancy on the reveal-state DAG, introduces a budget-constrained Frontier Reveal Marginalization estimator, and integrates it into a GRPO-style objective with detached token-score surrogates. Experiments on math and coding tasks show comparable or better results under matched budgets, with marginal gradient approximation improving as budget increases.

Introduction

Masked Diffusion Language Models generate text by iteratively unmasking positions, and have attracted increasing attention in text generation in recent years. Unlike autoregressive models that generate word by word, these models must select which positions to reveal from the model's own logits at each step, making the reveal order an endogenous latent variable with an extremely large discrete space. How to perform effective policy optimization for this characteristic is a key challenge in current research.

FEROM paper overview

Limitations of Existing Methods

Traditional policy optimization methods typically rely on pre-defined schedules or sampled trajectories to approximate the policy gradient. However, these methods either introduce bias or lead to high variance. Specifically, pre-defined schedules cannot adapt to the model's own learning dynamics, and although sampled trajectories are flexible, they have high variance that affects training stability.

Core Ideas of FEROM

To address the above issues, the paper proposes FEROM (Frontier Endogenous Reveal-Order Marginal Policy Optimization), an approximate policy gradient method for the marginal response policy induced by rollouts of masked diffusion language models. The key innovations of FEROM are:

  • Deriving a Rao-Blackwellized policy gradient identity with respect to the latent reveal order, expressing the estimator as posterior edge occupancy on the reveal-state DAG (directed acyclic graph).
  • Introducing Frontier Reveal Marginalization, a budget-constrained estimator that approximates the marginal edge structure by combining CDF-based local probability computation with frontier expansion of high-probability partial states.

Rao-Blackwellized Policy Gradient

Rao-Blackwellization is a classic variance reduction technique that reduces the variance of an estimator through conditional expectation. FEROM applies it to the latent variable of reveal order, thereby obtaining more stable gradient estimates. Specifically, FEROM expresses the policy gradient as the posterior occupancy probability of edges on the DAG, which allows us to exploit the graph structure for efficient computation.

Frontier Reveal Marginalization

To actually compute the marginalization, FEROM designs Frontier Reveal Marginalization. Under a given budget, this method computes local probabilities of the endogenous reveal process via the CDF and expands high-probability partial states to approximate the marginal edge structure. This budget-constrained estimator strikes a balance between computational efficiency and approximation accuracy.

Integration with GRPO

FEROM is integrated into a GRPO-style objective, replacing single-path log-scores with a detached token-score surrogate based on posterior-correlated reveal edges. This replacement enables the objective to better reflect the gradient of the marginal policy while keeping the implementation simple.

Experimental Evaluation

The paper conducts experiments on math and coding tasks. Under matched scoring-state budgets, FEROM achieves results comparable to or better than existing methods. In addition, an offline surrogate study shows that the marginal gradient approximation improves as the scoring budget increases. These results validate the effectiveness and scalability of FEROM.

Conclusion

FEROM provides a novel marginalization method for policy optimization of masked diffusion language models. Through Rao-Blackwellized policy gradients and frontier reveal marginalization, it effectively reduces variance and improves performance. Future work can further explore the application of this method to other generation tasks.

One-line Summary

FEROM proposes Rao-Blackwellized policy gradients and frontier reveal marginalization to reduce optimization variance for masked diffusion language models.

Paragraph Summary

This paper addresses the bias and high variance in policy optimization for masked diffusion language models caused by reveal order as an endogenous latent variable, and proposes the FEROM method. FEROM derives a Rao-Blackwellized policy gradient identity with respect to the latent reveal order, expresses the estimator as posterior edge occupancy on the reveal-state DAG, and introduces a budget-constrained Frontier Reveal Marginalization estimator that approximates the marginal edge structure through CDF-based local probability computation and frontier expansion of high-probability partial states. The method is integrated into a GRPO-style objective, replacing single-path log-scores with a detached token-score surrogate. Experiments on math and coding tasks show that under matched scoring-state budgets, FEROM achieves results comparable to or better than existing methods, and an offline surrogate study shows that the marginal gradient approximation improves as the budget increases.

#FEROM#masked diffusion language models#policy optimization#Rao-Blackwellization#GRPO#frontier reveal marginalization
Share:

Related articles