AI Video Generation Explained: Diffusion Models vs. Motion Transfer
Comparing diffusion models and motion transfer — the two core technologies behind AI video generation.
This article breaks down the two main technical approaches powering AI video generation: diffusion models (used by Sora, Runway, etc.) that create video from noise guided by text prompts, and motion transfer that extracts movement from source videos and applies it to new subjects. It compares their strengths, limitations, and ideal use cases, while exploring how hybrid approaches combining both are becoming the industry trend.
Why AI Video Generation Is So Fascinating
From Sora to Runway, from Pika to Kling, AI video generation technology has seen explosive progress over the past two years. Animation clips that once required professional teams spending weeks to produce can now be generated in minutes with a single text prompt. But what technology is actually driving these stunning results?
Current AI video generation relies on two fundamentally different technical approaches: Diffusion Models and Motion Transfer. Understanding the principles and differences between these two methods not only helps clarify the boundaries of the technology but also helps creators choose the right tools.
Diffusion Models: "Sculpting" Video from Noise
Core Principles of Diffusion Models
Diffusion models are the dominant technology in the current text-to-video space. Well-known products like Sora, Runway Gen series, and Kling are all built on this approach. The core idea is borrowed from image diffusion models (such as Stable Diffusion).
The training process of a diffusion model involves two steps: First, random noise is gradually added to real video frames until the image becomes pure "static"; then, a neural network is trained to learn how to reverse this noise removal process, step by step restoring the clear image. Once the model has learned this denoising process, it can start from pure random noise and, guided by text prompts, "generate" entirely new video content.
From a technical origins perspective, the mathematical foundation of diffusion models comes from stochastic differential equation theory in non-equilibrium thermodynamics. In 2020, Ho et al. proposed DDPM (Denoising Diffusion Probabilistic Models), which first demonstrated that diffusion models could match the image generation quality of the previously dominant GANs (Generative Adversarial Networks), and even surpass them in diversity and training stability. Subsequently, Stable Diffusion moved the diffusion process from pixel space to Latent Space, using a VAE encoder to first compress images into low-dimensional representations before denoising, dramatically reducing computational overhead and bringing diffusion models from academic labs to consumer applications. Video diffusion models build on this foundation by extending two-dimensional spatial generation to three-dimensional spatiotemporal generation — not only generating the spatial content of each frame but also ensuring temporal coherence between frames. The complexity grows exponentially, which is the core reason why video generation technology started much later than image generation.
The Challenge of Temporal Consistency
Unlike static images, video requires coherence between frames — objects can't suddenly deform, and character movements must be natural and fluid. This is the so-called "Temporal Consistency" problem.
To address this challenge, video diffusion models introduce Temporal Attention mechanisms, allowing the model to "reference" information from preceding and following frames when generating each frame. The recently emerging Diffusion Transformer (DiT) architecture (which is exactly the approach Sora uses) takes this further by decomposing video into spacetime patches, processing spatial and temporal dimensions in a unified manner, achieving longer and more coherent video output.
Looking deeper, the temporal attention mechanism is essentially an extension of the Transformer's Self-Attention along the temporal dimension. Traditional image diffusion models use U-Net as the backbone network, where convolution operations only perform feature extraction in the spatial dimensions, inherently lacking cross-frame modeling capability. To introduce the temporal dimension, early approaches inserted dedicated temporal attention modules between the attention layers of U-Net, enabling information exchange across different frames at the same spatial position. The DiT architecture, proposed by Peebles and Xie in 2023, completely replaced U-Net with a pure Transformer, splitting images or video into patch tokens for unified processing — naturally suited for introducing temporal information within the token sequence. OpenAI's Sora is based on a variant of the DiT architecture, compressing video into spatiotemporal patch sequences, enabling the model to learn long-range dependencies across frames — for example, after an object appears in frame 1, the model can still "remember" its appearance and position at frame 100. This architecture also demonstrates excellent scaling properties: as model parameters and training data grow, generation quality continues to improve, which is why the industry holds such high expectations for the DiT approach.
Motion Transfer: Bringing Existing Assets to Life
How Motion Transfer Works
Unlike diffusion models that "create from scratch," Motion Transfer takes a different route: it extracts motion information from a source video and applies it to a static image or character.
Typical use cases include: making a portrait photo perform dancing or talking movements following a reference video; or "transplanting" a dance routine onto a virtual character. Behind the scenes, this typically involves tracking and remapping of human body poses (skeletal keypoints), facial expressions, or Optical Flow.
On the technical implementation side, these motion representation methods each have their own focus. Optical Flow is a classic method in computer vision for describing pixel-level motion — it calculates the displacement vector of each pixel between two consecutive frames, forming a dense motion field that can precisely capture subtle movements like clothing flowing and hair swaying. Classic optical flow algorithms include Lucas-Kanade and Horn-Schunck methods, while in recent years, deep learning methods like RAFT have pushed optical flow estimation accuracy to new heights. Skeletal keypoint detection (such as OpenPose, MediaPipe, DWPose) is a sparse but semantically meaningful representation method that describes overall posture by locating 17 to 25 joint points on the human body. Each has its strengths and weaknesses: optical flow captures rich detail motion but is computationally expensive and extremely sensitive to occlusion; skeletal keypoints are more robust, easier to edit and retarget, but lose detail motion information like muscle contractions and clothing wrinkles. Modern high-quality motion transfer systems typically combine both — using skeletal keypoints to drive large-scale body movements, then using optical flow or deformation fields to supplement local details.
Advantages and Limitations of Motion Transfer
The biggest advantage of motion transfer is strong controllability and stable results. Since motion trajectories come from real reference videos, the generated results are often more natural and less prone to common diffusion model artifacts like "wrong number of fingers" or "flickering objects." This makes it highly popular in digital humans, virtual anchors, e-commerce model outfit changes, and similar applications.
However, its limitations are equally apparent: motion transfer is highly dependent on reference material and cannot truly "create" entirely new movements. Whatever action you want, you must provide a corresponding source video. This stands in stark contrast to the "unlimited imagination" capability of diffusion models.
Comparing and Merging Diffusion Models and Motion Transfer
Technology Selection Guide
| Dimension | Diffusion Models | Motion Transfer |
|---|---|---|
| Creative Freedom | High — can generate any content from text | Low — constrained by reference material |
| Result Controllability | Relatively weak | Strong |
| Temporal Consistency | Requires dedicated mechanisms | Naturally stable |
| Computational Cost | High | Relatively low |
| Typical Applications | Creative shorts, concept visuals | Digital humans, motion replication |
For scenarios pursuing creativity and originality, diffusion models are undoubtedly the better choice; for commercial applications requiring precise motion control and stable output, motion transfer is more practical.
Hybrid Approaches Are Becoming the Trend
These two technologies are not entirely opposed. An increasing number of cutting-edge solutions are beginning to merge them: using diffusion models to generate the base visuals and visual style, then using motion transfer or pose guidance (such as OpenPose in ControlNet) to precisely control motion trajectories. This "generation + control" combination is becoming standard in professional-grade AI video tools.
ControlNet, proposed by Zhang et al. in 2023, is a breakthrough method for injecting spatial conditional control into pretrained diffusion models. By copying the weights of the diffusion model's encoder and adding Zero Convolution layers, it enables various conditional signals such as depth maps, edge maps, and skeletal pose maps to precisely control generation results without destroying the original model's learned generation capabilities. This "plug-and-play" design philosophy is remarkably elegant — users don't need to retrain the entire diffusion model; they only need to train a lightweight control module. In the video domain, ControlNet's approach has been further extended to temporally consistent conditional control: users can provide frame-by-frame pose sequences to guide video generation, where each frame's skeletal pose determines the character's action, while the diffusion model handles filling in appearance, lighting, and background details. This architecture perfectly achieves an organic combination of diffusion model creativity and motion transfer controllability, making it possible to "precisely control every frame's action while maintaining high-quality visuals."
Conclusion: Choose the Right Technical Path Based on Your Needs
The pace of technological iteration in AI video generation is astonishing. Diffusion models are evolving toward longer duration, higher resolution, and stronger physical realism, while motion transfer is also continuously improving controllability by incorporating 3D modeling, neural rendering, and other technologies.
It's worth noting that Neural Rendering is becoming a key technology for improving motion transfer quality. Representative works include NeRF (Neural Radiance Fields) and 3D Gaussian Splatting, which can reconstruct freely viewpoint-renderable 3D scenes from multi-view images. Combining neural rendering with motion transfer means first reconstructing a person as a 3D representation, then applying skeletal driving in 3D space, and finally rendering back to 2D video. This approach achieves geometrically correct occlusion relationships and lighting effects, far superior to pure 2D image warping methods — for example, when an arm passes in front of the body, 3D methods can correctly handle front-back occlusion, while 2D methods often produce severe warping artifacts.
For content creators, understanding the fundamental differences between these two technical paths means being able to choose the right tools based on specific needs, or even combine them, finding the optimal balance between efficiency and quality. As technological boundaries continue to expand, AI video generation will unlock tremendous value across film, advertising, education, and many more fields.
Related articles
Random Pruning of the DPO Computation Graph: 1.4x Training Speedup and Lower Peak Memory
DPO training too slow and memory-hungry? This paper proposes a random pruning framework for the DPO computation graph, sampling independently across position, sequence, and branch dimensions; branch pruning achieves a 1.426x step-time speedup and cuts training time by about 30%, with in-depth attribution of memory and time gains.
Why Does JEPA Underperform MLM in Longitudinal EHR Representation Learning?
New study compares JEPA and MLM for longitudinal EHR representation learning on MIMIC-IV: three JEPA variants train stably without collapse but underperform MLM; target size, not heterogeneity, drives prediction error.
Transfer Learning + Visible Light Communication: A New Approach to Energy-Efficient Indoor Positioning
A study proposes a transfer learning-based visible light communication indoor positioning method, achieving 47% accuracy improvement on the BOSCH factory dataset, 67.5% model efficiency under high noise, and introducing the ME composite metric for energy-efficient positioning in energy-constrained industrial IoT scenarios.