Do Video Models Understand the World Better Than Image Models? Unpacking the World Model Debate

A Thought-Provoking Observation

Recently, an AI researcher posed a thought-provoking question on social media: "Why is this so much easier for AI than drawing a pelican? Could it be that video models have internalized better world models?"

What might seem like a casual remark actually touches on a core and profound issue in generative AI — to what extent do generative models across different modalities truly "understand" how the physical world works?

Understanding the Concept of World Models

World models are an important conceptual framework in AI, referring to an agent's internal abstract representation of how the external environment operates. A good world model should be able to predict: "If I take a certain action, how will the environment change?" The concept was first systematically articulated in the reinforcement learning domain, reaching a high point with David Ha and Jürgen Schmidhuber's 2018 paper World Models. Traditionally, world models required explicit design and training, but the emergence of large-scale generative models has opened new possibilities: Can models spontaneously acquire the ability to implicitly model the physical world through self-supervised learning on massive datasets? This is one of the central questions driving current research on video generation models.

The Origin of the "Pelican Test"

For those familiar with the AI evaluation community, "draw a pelican riding a bicycle" has become an informal but widely circulated benchmark test. It was originally proposed by developer Simon Willison as a quick way to assess the capability boundaries of large language models and image generation models.

This particular combination was chosen because it demands multiple competencies from the model: correctly understanding the anatomy of a pelican, grasping the mechanical structure of a bicycle, and then reasonably combining the two — figuring out how a bird would "ride" a vehicle designed for humans.

A Deeper Look at Compositional Generalization

This poses an extreme test of a model's compositional generalization ability — a key indicator for measuring an AI system's intelligence. Compositional generalization refers to whether a model can combine learned concepts in novel ways to understand and generate scenes never encountered during training. This is a core feature of human cognition — even if we've never seen "a purple giraffe driving a tank," we can immediately imagine a rough picture. But this is enormously challenging for neural networks. Current deep learning models tend to over-rely on statistical correlations in training data rather than truly understanding the independence and composability of concepts. The "pelican riding a bicycle" test became a classic precisely because it requires the model to genuinely understand "pelican" and "bicycle" as independent concepts and reasonably reason about their physical interaction — far beyond simple pattern matching.

As it turns out, the vast majority of models fail spectacularly at this task, producing all sorts of absurd images.

Why Video Models May Possess Stronger World Models

The original poster's core hypothesis is that video generation models may inherently possess stronger world models. This is far from baseless.

Physical Constraints Imposed by the Temporal Dimension

Image generation models only need to produce a single static, internally consistent frame. They can "cheat" — as long as a single frame looks plausible, there's no need to consider how objects move, how gravity acts, or how objects interact with each other.

Video models are an entirely different story. To generate a coherent video, the model must implicitly model temporal evolution: object trajectories must follow physical laws, occlusion relationships must remain consistent, and lighting changes must transition naturally.

The Technical Challenge of Temporal Coherence

This hard requirement for temporal coherence forces the model to learn dynamics representations that more closely approximate the real world during training. Temporal coherence is the core technical challenge in video generation, requiring that generated video sequences maintain physical consistency and semantic coherence across the time dimension. This includes: smoothness and predictability of object motion trajectories, correct handling of occlusions (objects should maintain their identity when they reappear after being occluded), continuous changes in lighting and shadows, and causal plausibility of interactions between objects. Early video generation models frequently exhibited "flickering," "jumping," and "objects suddenly disappearing or deforming" — all fundamentally symptoms of insufficient temporal coherence modeling. To address this, modern video models have introduced 3D convolutions, temporal attention mechanisms, optical flow constraints, and other techniques that force the model not only to generate plausible individual frames but also to ensure that transitions between frames align with physical intuition.

From Pixel Prediction to Dynamics Modeling

In other words, when a model is trained to predict "what happens in the next frame," it is essentially learning the causal structure of the world. This parallels the self-supervised paradigm in language models of "predicting the next token," but the density of physical information embedded in video data may be significantly higher.

Sora and Breakthroughs in Video Generation

This is precisely why, when video models like OpenAI's Sora and Google's Veo were released, the industry repeatedly emphasized that they are embryonic "world simulators." OpenAI's Sora, released in early 2024, represents a major leap in video generation technology. Unlike earlier video models, Sora employs a Diffusion Transformer (DiT) architecture that decomposes video into spatiotemporal patches for processing, capable of generating high-quality, coherent videos up to 60 seconds long. More importantly, OpenAI explicitly stated in its technical report that Sora is an embryonic "world simulator," claiming the model demonstrated understanding of 3D consistency, object permanence, and long-term temporal coherence. This sparked intense academic debate about whether video generation models truly model physical laws. Similar models include Google's Veo, Runway's Gen-3, and others — all evolving toward more precise modeling of the physical world.

Sora's technical report explicitly proposed that scaled video generation training is a promising path toward building a general-purpose simulator of the physical world.

Easier — or Just Appearing Easier?

However, this observation also warrants careful scrutiny to avoid over-interpretation.

Differences in Evaluation Standards

The reason the static image "draw a pelican" task seems so difficult is partly because the evaluation criteria are extremely strict — the human eye can stare at a still image for an extended period, picking out anatomical errors one by one. Videos, on the other hand, flow at dozens of frames per second, dispersing the viewer's attention. Many fine-grained flaws (such as finger counts suddenly changing or objects clipping through each other) actually go unnoticed.

Thus, the impression that video is "easier" may partly be a perceptual illusion rather than evidence that the model has truly mastered a better world model.

Differences in Training Data Scale and Quality

Another factor that cannot be ignored is data. Video models are often trained on massive amounts of real video footage — footage that is itself a "recording" of the physical world, naturally containing correct motion and interaction information. Models may primarily be "retrieving and recombining" real footage they've seen, rather than "reasoning" about physical laws from scratch. This point remains a subject of debate in current interpretability research.

Implications for AI Development and Embodied Intelligence

Regardless of the final conclusion, this discussion points to an increasingly clear trend: multimodal, temporally structured training data may be the key pathway toward stronger world models.

The Paradigm Shift Toward Embodied AI

If video models have indeed internalized better physical intuition, this carries profound implications for embodied AI, robotic control, autonomous driving, and other domains requiring precise world modeling. Embodied AI is an emerging paradigm in AI research that emphasizes intelligence must emerge and develop through interaction with the physical world, rather than merely processing abstract symbols. This idea challenges the traditional view that "intelligence equals symbolic reasoning," arguing that bodily perception and motor control form the foundation of higher cognition. In fields like robotics and autonomous driving, embodied AI requires AI systems not only to understand visual and linguistic information but also to accurately predict the effects of their actions on the environment and adjust strategies based on physical feedback.

A model that can accurately "imagine" physical consequences could conduct extensive trial and error in virtual environments, thereby reducing the cost and risk of real-world training. If video generation models truly acquire reliable world models, they could serve as "imagination engines" for embodied intelligence — robots could "imagine" the consequences of different actions in virtual environments, enabling safe and efficient planning and learning without dangerous and expensive real-world trial and error.

From the slightly whimsical test of "draw a pelican" to a serious discussion about the nature of world models — this exemplifies a unique culture in the AI community: using the most unassuming examples to probe the most profound questions.

Conclusion

Whether video models "understand the world" better than image models remains an open question, requiring more systematic controlled experiments to verify. But what is certain is that as models shift from processing isolated, static data to understanding continuous spatiotemporal processes, AI's grasp of the physical world is becoming increasingly nuanced. This may well be a quiet but important milestone on our journey toward artificial general intelligence.