WOVEN: Reframing Multimodal LLM World Model Training with Visual Transition Reasoning

Multimodal large language models repeatedly stumble on spatial, embodied, physical, and temporal reasoning, but most research treats these failures as independent weaknesses. A new paper offers a more unified explanation: they may all stem from a single missing capability—visual transition reasoning.

Around this hypothesis, the authors built a training data source and benchmark called WOVEN, and systematically validated its feasibility as a "shared training primitive."

A Hypothesis: All Failures Point to the Same Weakness

The paper's core claim is that MLLM failures on spatial, embodied, physical, and temporal tasks reflect an inadequate ability to model "how visual states change in response to actions." If this claim holds, then visual transition reasoning is not just a specialized skill for one task category, but a general capability that can be reused across models, supervision sources, and tasks.

To verify this, the authors needed an evaluation and training platform that could control variables: existing benchmarks either cover only a single scenario or test only a single reasoning operation, making controlled comparisons across scenarios, actions, and reasoning types impossible. WOVEN was created to fill this gap.

What is visual transition reasoning?

Visual transition reasoning refers to the ability to infer how the visual world will change to the next state, given a visual state (such as an image or a frame in a video) and an action or event applied to it. It encompasses both forward prediction ("if this happens, what will we see next") and inverse inference ("what must happen to return from the current state to the previous one"). This concept resonates with "intuitive physics" and "mental simulation" in cognitive science: when humans see a cup pushed off a table edge, they instinctively predict its falling trajectory without explicitly solving dynamic equations. MLLMs' difficulties on such tasks often stem not from a lack of static knowledge about the world, but from an inability to chain "observation" and "action" into a coherent state-transition sequence along the time axis.

WOVEN's Construction Approach

WOVEN organizes visual transition supervision along three dimensions: scenario, action, and reasoning type, with data drawn from diverse, realistic rollouts generated by a video-pretrained generative model. The final scale is:

• 36,076 samples
• 20 scenario types
• 5 action types
• 8 reasoning types

This three-dimensional annotation allows researchers to precisely answer "under what conditions does the model fail" and "what kind of training data can fill this gap," rather than piling up scores from dozens of benchmarks for spectacle.

Why use generative model rollouts as a data source?

WOVEN's visual transition supervision does not come from real video annotations, but from "rollouts" produced by a video-pretrained generative model—that is, sequences of future frames progressively inferred by the model given an initial frame and action conditions. The advantage is precise control over scenario types, action application methods, and reasoning targets, enabling the generation of large numbers of paired samples with clean labels, avoiding the problems of mixed actions and outcomes and high annotation costs in real videos. The trade-off is distribution shift: the generative model's own physical priors may carry biases or simplifications, making the data unable to fully reflect real-world complex dynamics. This is also an open problem acknowledged at the end of the paper—whether transition reasoning abilities trained on synthetic rollouts can robustly transfer to real scenarios still requires further validation.

38 Frontier Models, a Systematic Collective Deficiency

The authors evaluated 38 frontier MLLMs, including GPT-5.4 and Qwen3-VL-235B-A22B. The results were quite consistent:

• Even the strongest models fall far below human level;
• Failure modes recur across different model families;
• Simply scaling up model size does not eliminate these deficiencies.

This set of results reinforces the "shared weakness" hypothesis—if the problem were merely training data bias in one model family, consistent cross-family failures should not occur.

Leveraging ~2,000 Samples to Move 22 Benchmarks

More noteworthy are the training-side conclusions. The authors trained MLLMs at multiple parameter scales and found that the models learned a transferable general capability:

• Each training subset of about 2,000 samples collectively improved 22 out of 26 external benchmarks, with a maximum improvement of 27.3 percentage points;
• WOVEN data can replace 30%–50% of a task's own training data with comparable accuracy.

This means visual transition reasoning is not a marginal gain from "practicing one more type of problem," but more like a reusable foundational capability: small-scale, highly targeted supervision can produce measurable spillover effects across a large number of downstream tasks.

The relationship between MLLMs and "world models"

Multimodal large language models (MLLMs) refer to large-scale models capable of processing multiple modalities such as text, images, and video simultaneously, such as GPT-4V and Qwen-VL. The concept of a "world model" originates from reinforcement learning and cognitive science, referring to an agent's internal representation of external environment dynamics: given a current state and action, it can predict the next state and reward. WOVEN's argument essentially attributes some MLLM deficiencies to a lack of world model capability—that is, while models can describe image content, they cannot simulate its evolution with actions. This perspective explains why simply scaling parameters or increasing image-text pairs yields limited returns: descriptive knowledge can be scaled up, but dynamic modeling of state transitions requires dedicated supervision signals.

Training Recipe: Select Data by Reasoning Operation

The paper further provides an actionable training recipe and conducted prospective validation on held-out benchmarks:

1. Select data by the reasoning operation the supervision teaches, rather than by action, scenario, or domain. In other words, the value of training data depends on which type of reasoning it trains, not on how much it "looks like" the target task.
2. Prioritize samples with larger visual state changes for better robustness.

These two conclusions have direct implications for data engineering: organizing training sets by difficulty, domain, or surface similarity may not be as effective as organizing them by reasoning operation type.

Why This Matters

If visual transition reasoning is indeed a transferable shared capability, then the training approach for multimodal world models needs adjustment: instead of piling up data separately for each downstream task, first establish general modeling of visual state changes, then transfer to specific tasks.

WOVEN's value lies in providing both evaluation standards and a training recipe, with the recipe validated on benchmarks not involved in training. This is more convincing than releasing only a dataset or reporting only a set of scores. Of course, whether the rollouts provided by the generative model can fully cover real-world complex dynamics remains an issue requiring ongoing observation.