GraftOOD: Solving the "Motion Collapse" Problem in Video Reward Optimization

What Is "Motion Collapse" in Video Generation?

During reinforcement learning fine-tuning of video generation models, researchers commonly use reward optimization to improve generation quality. However, the GraftOOD study reveals a long-overlooked problem: when a model single-mindedly chases improvements in proxy reward, the dynamism of the generated video actually degrades. The research team named this phenomenon "motion collapse."

What makes this even more alarming is that even when a video-level motion-quality reward is introduced, the collapse still occurs. This suggests there is an inherent mismatch in relying purely on scalar reward signals to guide video generation — the model learns to game the reward function while sacrificing the motion coherence and realism that video should naturally have.

Reward optimization in video generation typically borrows from the RLHF paradigm in large language models: a proxy reward model is trained to evaluate generation quality, and that reward signal then drives updates to the generative model. A proxy reward is fundamentally an approximate estimate of true human preference, and this approximation inherently carries the risk of "reward hacking" — the model may learn to maximize reward scores without genuinely improving perceptual quality. In the video domain, this risk is especially pronounced: compared to static images, the temporal coherence of motion is difficult to fully capture with a single scalar score. "Motion collapse" is the archetypal manifestation of this mismatch, where the model tends to generate low-dynamic or near-static frames, since static video often appears "more stable and sharper" to perceptual reward models and thus receives higher scores.

GraftOOD's Core Idea: Grafting Out-of-Distribution Samples

The name GraftOOD comes from the act of "grafting." The approach works by replacing a portion of online rollouts with out-of-distribution (OOD) video exemplars within selected prompt groups.

The elegance of this design lies in how it grafts external supervision signals into a reinforcement learning pipeline that would otherwise rely entirely on self-sampling. Instead of learning only from its own generated samples, the model can reference high-quality external exemplars to correct its direction, thereby avoiding getting trapped in local optima of the reward function.

Shared Velocity Regression Objective

On the technical side, GraftOOD employs a shared velocity-regression objective that combines two types of supervision signals:

• Group-relative reward rectification: Continues the reward-weighting approach based on group advantage estimation;
• Sequence-level exemplar supervision: Introduces direct supervision from external exemplars.

This design has two practical engineering advantages: it requires neither additional exemplar generation trajectories nor extra supervision loss coefficients to balance the two types of losses. The research team provides a theoretical characterization of the method's pointwise optimal field, describing it as a "source-posterior-weighted combination" of two specific source objective fields — giving the approach an interpretable mathematical foundation.

In online RL, the model continuously samples from its current policy (online rollouts) and updates its parameters accordingly. This bootstrapped learning can cause distribution drift — the model sees only its own generated samples and progressively diverges from the real data distribution. Out-of-distribution (OOD) samples come from outside the current policy's distribution; they can be pre-collected high-quality videos or outputs from other models. The intuition behind incorporating OOD samples is that when a model falls into a local optimum (e.g., generating static frames to inflate reward scores), dynamic video exemplars from outside the distribution provide "anchor points" that break the self-reinforcing negative cycle. This is analogous to using a reference model in DPO (Direct Preference Optimization) to impose KL-divergence constraints — except GraftOOD performs the substitution directly at the sampling level rather than implicitly through KL divergence.

Velocity regression is a core training objective in diffusion/flow matching video generation models: the model learns to predict the "velocity field" from noise to clean video — i.e., the direction of each denoising step. GraftOOD's choice to unify both supervision signals at the velocity field level, rather than at the final generated frame level, is significant: since the velocity field is the model's direct output, applying supervision there is both precise and efficient, avoiding the extra overhead of re-noising and decoding exemplar videos. Group-relative reward rectification follows the GRPO (Group Relative Policy Optimization) approach, estimating the advantage function through the relative quality of within-group samples without needing a separate value function network, reducing training complexity. Since both objectives are naturally unified within the same velocity regression framework, no additional loss-weight hyperparameter is needed for manual balancing.

The GraftSet-Video Dataset

To provide external supervision for the model, the research team constructed a dedicated dataset: GraftSet-Video. It contains 2,397 prompts corresponding to 9,588 format-matched video exemplars, averaging approximately 4 exemplars per prompt.

Format matching is a critical detail — external exemplars must remain format-consistent with the model's own generated samples to be seamlessly grafted into the training pipeline without introducing excessive distributional noise.

Experimental Results: Dramatic Improvements in Dynamism

Experiments were conducted on the Wan2.1-1.3B model, yielding compelling results. Compared to the reward-only baseline FlowAWR:

• Dynamic Degree jumped from 4.86 to 44.41 — nearly a 9× improvement, directly targeting the motion collapse problem;
• HPSv3 declined slightly from 8.10 to 7.32;
• VisionReward remained roughly unchanged.

Under the primary experimental setup, GraftOOD led on nine out of ten metrics in a head-to-head comparison among three exemplar augmentation methods. This demonstrates that introducing external exemplar supervision does not trade overall quality for dynamism — rather, it achieves a relatively balanced improvement.

Dynamic Degree is a commonly used evaluation dimension in video generation benchmarks such as EvalCrafter and VBench, quantifying motion intensity via optical flow magnitude or inter-frame differences — higher values indicate more vigorous motion. HPSv3 (Human Preference Score v3) is an aesthetic scoring model trained on human preference annotations that tends to reward clear, compositionally stable frames. This partly explains why HPSv3 dips slightly as dynamism increases — more motion means more blur and complexity, which don't always score well on static aesthetic dimensions. This phenomenon also corroborates the paper's warning that "high dynamic degree ≠ high-quality motion": a single metric cannot comprehensively measure video motion quality; a multi-dimensional evaluation framework is needed.

Trade-offs Revealed by Ablation Studies

The research team conducted ablation analyses on exemplar frequency and within-group exemplar count, arriving at an important conclusion: increasing exemplar usage does not yield consistent gains; instead, it presents a trade-off between dynamism and reward.

This finding is admirably restrained and honest. It reminds us that external exemplar supervision is a double-edged sword — too much may over-prioritize dynamism at the expense of other quality dimensions; too little fails to effectively mitigate motion collapse. Finding the right balance is a key tuning challenge for practical applications.

Implications for the Field

GraftOOD's value lies not only in proposing a concrete method, but also in its reflection on the video reward optimization paradigm. The research explicitly states: external exemplar supervision should serve as a complement to scalar reward feedback, not a replacement.

The paper also specifically emphasizes that one should not simply equate "higher measured dynamism" with "coherent, visually plausible motion." In other words, while the dramatic improvement in Dynamic Degree is encouraging, the numbers alone cannot fully represent perceptual motion quality as judged by the human eye. This clear-eyed awareness of evaluation metric limitations is particularly valuable in today's research climate, where metric competition runs rampant.

The research code has been open-sourced on GitHub (under the project name FlowAWR), making it easy for subsequent researchers to reproduce and extend. For researchers working on video generation and reinforcement learning fine-tuning, GraftOOD offers a worthwhile technical path: when pure reward optimization hits a dead end, judiciously introducing high-quality external exemplars may be the key to breaking through.