DEEPO: Dual-Entropy Enhanced Policy Optimization Tackles Hallucination in Multimodal LLMs

Multimodal large language models (MLLMs) have made remarkable strides in reasoning, thanks in no small part to reinforcement learning (RL). Yet an inconvenient truth remains: RL's ability to suppress hallucinations is inconsistent — sometimes effective, sometimes not. A new arXiv paper titled DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs zeroes in on the "correction chain" between rewards and parameter updates, proposing a two-stage enhancement framework.

Why RL Inconsistently Suppresses Hallucinations

The paper's core insight is that existing RL methods based on group-relative advantage have two weak links in the correction chain — and both occur precisely where hallucination risk is highest.

The first weak point appears during the rollout (sampling) stage. For "hard questions" with high semantic entropy, the model tends to generate an entire group of incorrect samples. When all samples in a group are wrong, the group-relative advantage collapses to zero. In other words, in high-uncertainty scenarios where learning signals are most needed, the model receives no useful gradient feedback whatsoever.

The second weak point lies at the optimization level. So-called "confident-but-wrong" tokens are effectively gradient-invisible: for a categorical policy, as the output distribution becomes increasingly sharp, the expected score-gradient norm approaches zero. This means the predictions the model is most certain about — yet most wrong about — receive the weakest updates. This is one of the deep mechanisms behind hallucination's persistence: the model is overconfident in its wrong answers, and gradients can't touch them.

Semantic Entropy is a measure of the semantic diversity of model outputs, proposed by Kuhn et al. in 2023. Unlike ordinary token-level entropy, semantic entropy clusters outputs with similar meanings into the same semantic category and computes entropy over those categories. A question with high semantic entropy means the model's "interpretation directions" are highly scattered — not just different phrasings, but semantically divergent intents. Such questions typically involve ambiguous visual information, open-ended prompts, or fuzzy knowledge boundaries, which are exactly the scenarios where multimodal models are most prone to hallucination. Group-Relative Advantage is the core mechanism behind methods like GRPO: multiple outputs are sampled for the same question, and each output's relative advantage is computed against the group's mean reward, replacing the hard-to-estimate value function in traditional RL. This requires reward variance within the group; once all samples share the same reward (e.g., all zero), advantages collapse to zero and gradient signals vanish.

DEEPO's Dual-Branch Design

To address these two pain points, the authors propose Dual-Entropy Enhanced Policy Optimization (DEEPO), a two-stage enhancement combining signal-variance regularization and gradient preconditioning.

Semantic-Entropy-Triggered Expert Prefixes

The first branch addresses the advantage-collapse problem at the rollout level. When a query's semantic entropy is detected to be too high, the system injects expert prefixes — factually grounded, guided continuations. This effectively provides a supervised signal on high-uncertainty questions, restoring variance in the advantages. With variance restored, the model can once again obtain gradient differences useful for learning, avoiding a "dead zone" on hard questions.

Expert Prefixes are essentially a conditional supervised injection mechanism. In standard RL sampling, the model generates responses entirely from its own distribution; with expert prefixes injected, the model's subsequent generation is "anchored" to a factually correct starting point, dramatically increasing the probability of correct answers appearing within the sampled group. This bears some resemblance to Rejection Sampling Fine-tuning or chain-of-thought distillation, but the key difference is that it is dynamically triggered — activated only when semantic entropy exceeds a threshold, rather than applied uniformly to all questions. This avoids over-compressing the model's exploration space. The design is also sound from an engineering perspective: for low-entropy "easy questions," the model can already sample correct answers on its own, so forcing prefix injection would introduce bias; intervening only on "hard questions" is a precise, minimal-cost restoration of the learning signal.

Advantage-Sign-Aware Renyi Preconditioning

The second branch targets logit saturation at the optimization level. The authors introduce advantage-sign-aware Renyi preconditioning to counteract saturation at the logit level. Its goal is to ensure that correction signals can actually reach "confident-but-wrong" predictions — effectively redirecting gradients toward the places most in need of correction within the operative confidence regime.

Both branches independently improve upon the GRPO baseline, and their synergy is especially pronounced on the most complex tasks.

Renyi preconditioning draws from information geometry and natural gradient methods. Standard gradient descent measures update steps in Euclidean parameter space, while preconditioning methods rescale gradients using curvature information (e.g., the Fisher information matrix) to make update steps more uniform in probability distribution space. Renyi divergence is a generalization of KL divergence, and its order parameter can be tuned to control sensitivity to distribution tails or peaks. In logit-saturated scenarios, a token's output probability is already extremely close to 1, and the Euclidean gradient norm approaches zero — yet there is still meaningful "room to move" in distribution space. Renyi preconditioning re-parameterizes this curvature relationship so that gradient signals remain effective even in high-confidence regions, allowing "confident-but-wrong" predictions to be genuinely penalized. Advantage-sign awareness means the preconditioning direction is adjusted according to the sign of the advantage for the current token, ensuring correct predictions are reinforced and wrong ones are suppressed — rather than applying the same intervention uniformly to both.

Experimental Results and Significance

On VideoMMMU — the most complex long-horizon task in the evaluation suite — the paper observes a statistically significant interaction effect between the two branches: a +4.0 improvement over the baseline (95% CI [1.1, 6.9]). On other tasks, the two branches' contributions are additive — they don't cancel each other out.

Critically, DEEPO reduces hallucinations while maintaining both model accuracy and training stability. This matters: many hallucination-targeting interventions come at the cost of overall performance or training stability, whereas DEEPO claims to achieve all three simultaneously.

From a methodological standpoint, DEEPO's value lies not just in its two specific techniques, but in how it decomposes the fuzzy phenomenon of "hallucination" into two mathematically tractable, intervenable points — advantage collapse and gradient invisibility. This approach of "tracing the correction chain to find the leaks" provides a clear analytical framework for future multimodal RL alignment research.

Summary

DEEPO reminds us that hallucination is not simply a matter of "insufficient data" or "models not being large enough" — it may be a structural defect in training dynamics. When learning signals for high-uncertainty questions are wiped out, and when confident-but-wrong predictions slip through gradient updates, no amount of additional RL training can reach the root cause of hallucination. Through the one-two punch of semantic-entropy-triggered supervised injection and Renyi gradient preconditioning, DEEPO offers a technically compelling path toward improving the reliability of multimodal large language models.