How AI Video Generation Works: Diffusion Models, Motion Transfer, and Optical Flow Explained

Introduction: Why AI Video Generation Is So Complex

Over the past two years, AI image generation has experienced explosive growth — from Stable Diffusion to Midjourney, the quality of static image generation has become convincingly realistic. However, AI video generation remains a far more daunting challenge. The reason is simple: video requires not only that each individual frame looks realistic, but also that frames maintain temporal consistency. An object can't suddenly deform, flicker, or vanish in the next frame.

To grasp the scale of this challenge, consider a simple calculation: a 4-second video at 24 fps contains 96 frames, and each frame must maintain consistent geometric structure, lighting conditions, and object identity. This means the model needs not only the ability to generate high-quality individual frames, but also to maintain global coherence constraints across an extremely high-dimensional spatiotemporal space — orders of magnitude harder than image generation alone.

A recently circulated visual tutorial from Reddit's tech community breaks down the core mechanisms of AI video generation — from Diffusion to Motion Transfer to Optical Flow. This article systematically explores the technical logic behind AI video generation based on these concepts.

Diffusion Models: The Core Engine for Restoring Images from Noise

Fundamental Principles

Diffusion models are the foundation of current mainstream AI generation technology. The core idea is borrowed from physical diffusion processes: during training, the model learns to gradually add random noise to a clear image until it becomes pure noise; during generation, it reverses this process — starting from pure noise and progressively denoising step by step to produce an image matching a text description.

The theoretical foundation traces back to non-equilibrium thermodynamics. In the physical world, a drop of ink in water gradually diffuses until evenly distributed — an irreversible process. But diffusion models cleverly leverage stochastic differential equation (SDE) theory to prove this process can be mathematically reversed. Specifically, the forward process follows a predefined noise schedule, typically using Gaussian noise, while the reverse process trains a parameterized neural network (usually a U-Net or the more recent DiT architecture) to learn the denoising function at each step. This framework was first proposed by Sohl-Dickstein et al. in 2015, later popularized by DDPM (Denoising Diffusion Probabilistic Models) in 2020, and significantly accelerated through sampling methods like DDIM that reduced the number of inference iterations.

This process typically requires dozens of iterative steps, with a neural network predicting how much noise to remove at each step. This gradual restoration mechanism is what enables diffusion models to generate highly detailed and diverse content.

From Image Diffusion to Video Diffusion Models

Extending diffusion models to video generation most directly involves having the model process multiple frames simultaneously. Video diffusion models introduce a temporal dimension beyond the spatial dimensions (width, height), enabling the network to understand inter-frame relationships while denoising.

These models typically incorporate temporal attention layers, allowing each frame's generation to reference information from neighboring frames, ensuring smooth temporal transitions. Temporal attention layers are the core component distinguishing video diffusion models from image diffusion models. In standard image diffusion models, attention operates only in the spatial dimension — correlating different positions within a single frame. Temporal attention layers enable information exchange across different frames at the same spatial position. This is typically implemented by rearranging the video tensor so the temporal dimension becomes the sequence dimension, then applying standard Multi-Head Self-Attention. Representative architectures like Google's Video Diffusion Models and Meta's Make-A-Video commonly use alternating "spatial attention + temporal attention" stacking designs. Some approaches like AnimateDiff adopt a lighter strategy — training only pluggable temporal attention modules on top of a frozen image diffusion model, achieving video generation capability at lower computational cost.

However, relying purely on diffusion for coherence is extremely expensive and prone to detail flickering. This leads us to two other key technologies.

Motion Transfer: The Key to Animating Static Characters

What Is Motion Transfer?

Motion transfer refers to transferring motion from a source video onto a different target object. For example, given a reference video of someone dancing and a static character image, motion transfer technology can make the static character "learn" the dance moves from the reference video.

Notable examples include the early First Order Motion Model (FOMM) and recent diffusion-based animation tools. FOMM, proposed by Aliaksandr Siarohin et al. in 2019, was a milestone in motion transfer. Its key innovation was using first-order Taylor expansion to approximate local affine transformations around keypoints, enabling realistic motion transfer without 3D modeling. The model consists of three modules: a keypoint detector, a dense motion network, and an image generator. The team later proposed the Thin-Plate Spline Motion Model, replacing affine transformations with thin-plate spline transforms to support more complex non-rigid motion. In 2023-2024, diffusion-based motion transfer approaches like Animate Anyone, MagicAnimate, and LivePortrait elevated generation quality to new levels, typically combining pose estimation tools like DensePose or OpenPose for more precise human motion representation and leveraging pretrained diffusion models' strong priors for more natural animation.

These models typically first extract motion representations from the reference video — such as keypoints, skeletal poses, or motion fields — then use this motion information as conditioning to drive frame-by-frame deformation of the target image.

Motion-Appearance Disentanglement

The key to successful motion transfer lies in disentanglement — separating motion information from appearance information. The model must understand what constitutes motion (to be extracted from the reference video) and what constitutes identity and appearance (to be preserved from the target image).

Disentangled Representation Learning is a long-studied topic in machine learning. The goal is for different dimensions of the model's learned latent representation to correspond to different semantic factors that can vary independently. In the context of motion transfer, disentanglement means decomposing video information into identity-related static features (e.g., facial structure, clothing texture) and motion-related dynamic features (e.g., pose changes, expression changes). Common disentanglement strategies include: information bottleneck constraints (preventing the motion encoder from carrying appearance information), contrastive learning (ensuring encodings of different identities with the same motion are close in motion space), and cross-reconstruction (reconstructing video using A's appearance and B's motion, then verifying disentanglement quality through reverse reconstruction). Disentanglement quality directly determines motion transfer fidelity — incomplete disentanglement causes the target character to "leak" appearance features from the source.

Only by achieving this can we get effects like "swap the face but keep the motion" or "swap the motion but keep the person." This is why many motion transfer models feature dedicated motion encoders and appearance encoders.

Optical Flow: The Bridge Between Frames

The Role of Optical Flow in Video Generation

Optical Flow is a classic concept in computer vision that describes the motion vector of each pixel between two adjacent frames. Simply put, optical flow tells us "where did this pixel move from the previous frame to the next."

The concept was first proposed by psychologist James Gibson in 1950 to describe the apparent motion patterns of visual scenes as an observer moves through an environment. In computer vision, classic optical flow algorithms include the Lucas-Kanade method (local constraint-based) and the Horn-Schunck method (global constraint-based), both relying on the brightness constancy assumption — that pixel brightness of the same object remains unchanged across adjacent frames. In the deep learning era, FlowNet (2015) was the first to estimate optical flow end-to-end using CNNs, followed by RAFT (Recurrent All-Pairs Field Transforms, 2020), which dramatically improved accuracy by constructing all-pairs correlation volumes with iterative refinement, becoming one of the most widely used optical flow estimation methods.

In AI video generation, optical flow plays a crucial role. By estimating optical flow, models can warp information from the previous frame to the current frame, introducing motion while maintaining content consistency. The warp operation is expressed as: I_{t+1}(x) = I_t(x + F_{t→t+1}(x)), where F is the flow field. However, warping has inherent limitations: when occlusion occurs, some regions have no corresponding source information in the next frame; when dis-occlusion reveals new content, warping cannot create imagery from nothing. Therefore, modern systems typically employ occlusion detection masks and inpainting networks to handle warp failure regions, or use warp results merely as guidance conditions for diffusion models rather than final output.

Many video generation and editing approaches leverage optical flow to constrain results, preventing flickering and content drift.

Combining Optical Flow with Diffusion Models

Integrating optical flow into the diffusion generation pipeline is an effective approach for improving temporal coherence. A common strategy is to first generate keyframes using the diffusion model, then use optical flow for interpolation and guidance between keyframes, constraining intermediate frame generation with motion information.

Keyframe Interpolation is an important engineering strategy for balancing quality and efficiency in current video generation systems. The basic idea: first generate a small number of keyframes (e.g., 2-4 per second) using a computationally intensive but high-quality diffusion model, then fill in intermediate frames with a lightweight interpolation model. During interpolation, optical flow provides motion constraints ensuring intermediate frames maintain consistent motion trajectories with keyframes. Representative works include FILM (Frame Interpolation for Large Motion) and AMT (All-Pairs Multi-Field Transforms). Some commercial products adopt similar hierarchical generation strategies. Additionally, approaches like SVD (Stable Video Diffusion) explore temporal interpolation in latent space, further reducing computational overhead. This layered architecture makes generating videos of several seconds or longer feasible, rather than being limited to extremely short clips.

This leverages both the powerful generation capability of diffusion models and the physical plausibility of motion ensured by optical flow, forming a complementary architectural design.

How the Three Technologies Work Together

As the above breakdown shows, modern AI video generation typically doesn't rely on a single technology, but rather a combination of mechanisms:

Diffusion models generate high-quality, diverse visual content
Motion transfer applies reference motions to target objects for controllable dynamics
Optical flow establishes pixel-level correspondences between frames to ensure temporal coherence

Together, these three address the core tension in video generation between "visual quality" and "temporal consistency." Understanding their division of labor helps us see the technical differences behind various AI video tools (such as Runway, Pika, Sora, etc.).

Notably, OpenAI's Sora, released in early 2024, represents a new direction. Unlike previous U-Net-based video diffusion models, Sora adopts the Diffusion Transformer (DiT) architecture, treating video as a sequence of spatiotemporal patches — similar to how LLMs process tokens. This architecture naturally supports variable resolution and duration, and exhibits clear Scaling Law characteristics as model size increases. Another key innovation is operating in a compressed latent spatiotemporal space, mapping raw pixel-space video to low-dimensional representations via a video compression network, dramatically reducing computational requirements. While Sora's full technical details remain unpublished, its demonstrated ability to generate one-minute high-quality video suggests that sufficient model scale combined with carefully designed training data strategies can largely mitigate temporal consistency issues, reducing dependence on explicit constraints like optical flow. This also indicates that the weighting and combination of these three technologies continues to evolve as model capabilities improve.

Conclusion: The Long-Term Value of Understanding Fundamentals

This Reddit visual tutorial gained attention precisely because it transforms otherwise abstract mathematical processes into intuitive diagrams. For beginners, understanding "how noise is gradually restored," "how motion is extracted and transferred," and "how pixels flow between frames" is far more effective than memorizing formulas.

As technology rapidly evolves, the barrier to AI video generation continues to lower, but the underlying principles become increasingly worth understanding deeply. Mastering these three pillars — diffusion models, motion transfer, and optical flow — helps creators and developers better evaluate tools, optimize results, and build a solid foundation for further research and practice.