Visually Attributable Self-Distillation: VAG, a New Paradigm for Multimodal RL

Multimodal reinforcement learning is facing a hidden pitfall in visual reasoning tasks: dense token-level corrections from a "privileged teacher" may be mixed with non-visual interference, which instead reinforces erroneous updates. A recent study proposes a visually attributable on-policy self-distillation framework and uses a Visual Attribution Gate (VAG) to keep contaminated corrections out, surpassing GRPO and OPSD on six visual reasoning benchmarks.

The Problem: Why Dense Corrections Can Hurt

Multimodal reinforcement learning with verifiable rewards (RLVR) provides task-level supervision—whether an answer is right or wrong is clear, but the model has difficulty knowing which step in the reasoning chain went wrong. On-policy self-distillation (OPSD) attempts to fill this gap: it introduces a privileged teacher that possesses an "evidence-centered visual view" and provides token-level dense corrections to the student model.

The problem is that these corrections are not pure. The study formalizes this phenomenon as privileged correction contamination: when the direction of a token-level update cannot be consistently attributed to privileged visual evidence, that correction is contaminated. Sources of contamination may include language priors, answer style, or even non-visual shifts introduced by the privileged view itself. Directly distilling contaminated corrections amounts to treating noise as a supervision signal, which may in turn reinforce erroneous token-level updates.

On-policy self-distillation (OPSD) is a variant of knowledge distillation: instead of relying on a fixed offline teacher model, it generates supervision signals in real time from the current policy (or an enhanced version of it) during training and passes them back to the student model at token-level granularity. Compared with RLVR rewards that only indicate whether an answer is right or wrong, OPSD can provide denser gradient signals and alleviate the credit assignment problem under sparse rewards. In visual reasoning, the teacher is usually endowed with additional visual information (such as high-resolution crops or evidence-annotated views), thereby playing the role of a "privileged teacher." But precisely because the teacher sees information the student cannot, its correction directions may incorporate non-visual factors—this is the structural root of the contamination problem.

Privileged Information Distillation originates from the "Learning Using Privileged Information" (LUPI) framework in learning theory: during training, the teacher is allowed to access additional features (such as bounding boxes, depth maps, or expert reasoning chains), while at deployment the student can rely only on standard inputs. This paradigm can significantly improve sample efficiency, but it carries an implicit assumption—that the teacher's extra information is indeed relevant to the task. If the privileged view itself is biased (for example, over-cropping leads to missing context), the token-level preferences given by the teacher may reflect view artifacts rather than genuine visual evidence. In multimodal settings, the strength of language priors further amplifies this shift, causing "plausible-looking" corrections to actually teach the model to take shortcuts.

The Method: "Verifying" Corrections with Visual Contrast

The evidence-consistency paradigm proposed by the study has a core idea: every privileged correction must pass a test of visual evidence. This is implemented as the Visual Attribution Gate (VAG), whose mechanism contains two key designs.

Using verifier advantage as the optimization anchor. VAG does not abandon the task-level signal of RLVR; instead, it uses it as the baseline anchor for the entire optimization, ensuring that the overall direction of policy updates does not deviate from verifiable correctness.

Gating by comparing evidence-present and evidence-degraded views. For each privileged correction, VAG constructs two teacher views—evidence-present and evidence-degraded—and observes whether the correction direction receives directional support in this visual contrast. Only corrections consistently supported by the visual contrast are accepted, while the remaining inconsistent residuals are suppressed. This preserves visually grounded guidance while filtering out confounding factors.

The ingenuity of this design lies in turning "is this correction really about vision?" into an operational contrastive test rather than relying on heuristic assumptions.

GRPO (Group Relative Policy Optimization) is one of the mainstream algorithms in current RLVR: it samples a group of responses to the same question and normalizes using the mean and standard deviation of rewards within the group to obtain the relative advantage of each response, thereby eliminating the need for a separate critic network. Its advantages are simple implementation and no value model, but the supervision granularity remains sequence-level—the entire response shares one advantage value and cannot indicate which step in the reasoning chain went wrong. VAG and GRPO are not substitutes: the former uses the GRPO-style verifier advantage as the optimization anchor and, on top of that, adds token-level attributable corrections, which amounts to fine-grained credit assignment under a coarse-grained correctness constraint.

Experiments: Consistent Lead on Six Benchmarks

The study validated the method on six visual reasoning benchmarks. The results show that VAG achieves consistent advantages over both GRPO and OPSD methods and sets a new best average performance. The paper emphasizes that its advantage is "consistently"—that is, stably reproduced across multiple benchmarks rather than being a single-point breakthrough.

For practitioners of multimodal RL, this result sends a clear signal: when introducing a privileged teacher for dense supervision, the purity of the source of correction signals matters as much as the signals themselves.

Visual Attribution in interpretability research refers to methods that attribute model outputs to specific image regions or visual features, with common approaches including gradient saliency, attention rollout, and perturbation contrast. VAG borrows the idea of "perturbation contrast": by constructing evidence-present and evidence-degraded views, it observes whether the correction direction changes consistently with the increase or decrease of visual evidence. If a correction direction remains unchanged after evidence degradation, it is driven by language priors or stylistic factors rather than visual grounding. This "counterfactual consistency test" turns attribution from an offline analysis tool into an online gating signal during training, which is the core difference between this method and traditional interpretability techniques.

Implications: Dense Supervision Needs Provenance Auditing

The value of VAG goes beyond a single gating module. It brings the meta-question of "where corrections come from" into the optimization loop of multimodal reinforcement learning. When teacher views, language priors, and answer style all act on token-level signals simultaneously, self-distillation without an attribution mechanism can easily amplify bias without being noticed. The Visual Attribution Gate provides a lightweight but principled auditing approach—letting evidence comparison speak, so that every correction can be traced back to visual grounding.

For teams building visual reasoning systems, this means that when designing distillation pipelines, "attributability of corrections" should be listed as a design goal on par with "density of corrections."

One-line Summary

Using a Visual Attribution Gate to filter contaminated corrections from a privileged teacher, improving visual reasoning performance in multimodal reinforcement learning.

Paragraph Summary

The article introduces a study on visual reasoning tasks in multimodal reinforcement learning. The study points out that token-level dense corrections provided by a privileged teacher in on-policy self-distillation may be mixed with non-visual factors such as language priors and answer style, forming "privileged correction contamination"; direct distillation would in turn reinforce erroneous updates. To address this, the study proposes a visually attributable on-policy self-distillation framework, whose core is the Visual Attribution Gate (VAG): using verifier advantage as the optimization anchor and comparing evidence-present and evidence-degraded teacher views to test whether each correction direction is consistently supported by visual evidence, accepting only corrections that pass the test. On six visual reasoning benchmarks, the method consistently surpasses GRPO and OPSD and sets a new best average performance, with the implication that dense supervision needs built-in attributability auditing.