Video Diffusion Models Power Controllable 3D Avatars: VDAs Break the Single-Image Generation Bottleneck

Video diffusion models (VDMs), trained on internet-scale data, have already acquired rich priors over appearance, motion, and even physical laws—advantages that traditional explicit 3D rendering and expensive 3D data capture are hard-pressed to replicate. A new paper proposes Video Diffusion Avatars (VDAs), which leverages this prior to generate photorealistic, multi-view consistent, controllable avatars from a single image.

From 3DMM to a Learned Expression Space

Traditional avatar methods mostly rely on 3D Morphable Models (3DMMs) to drive expressions, but 3DMMs have a ceiling on expressiveness, especially in regions like the tongue and the interior of the mouth. VDAs takes a different path: replacing 3DMM with a learned expression space, thereby breaking through the expressiveness ceiling.

However, the paper highlights a key problem: learned expression encodings suffer from severe view bias. The authors build the DISTRACT benchmark to quantify this phenomenon and demonstrate that disentanglement can eliminate the bias, enabling these encodings to serve as driving signals for multi-view consistent avatars.

3DMM (3D Morphable Model) is a classic parametric face representation proposed by Blanz and Vetter in 1999. Its core idea is to apply PCA to a large set of 3D face scans to obtain a set of low-dimensional basis vectors, so that facial shape and texture variations can be described with a small number of coefficients. 3DMM's advantages are its linearity, interpretability, and stable driving; it remains the underlying backbone of avatar systems such as Face2Face, DECA, and EMOCA to this day.

Its limitations are also clear: as a linear model, it can only represent deformation patterns present in the training set; regions such as the tongue, gums, and mouth interior are invisible or unlabeled in most scan data, so 3DMM can hardly produce plausible deformations there. This is precisely the direct motivation for the paper's proposal to replace 3DMM with a nonlinear learned expression space.

View bias refers to the fact that when an expression encoder is learned from monocular video, the encoding results inadvertently absorb information about camera viewpoint, lighting, and head orientation, rather than pure expression. In downstream tasks, this manifests as the same expression being encoded into different vectors under different shooting angles, causing the driven avatar to jitter or deform across views.

This is similar to "shortcut learning" in representation learning—networks tend to exploit factors in the data that are correlated with labels but not causal. The paper's solution is disentanglement: letting the encoding channel carry only expression semantics, while handing view-related information to explicit camera parameter (proxy-mesh rendering) channels. The role of the DISTRACT benchmark is to turn this previously neglected problem into a quantifiable, comparable metric.

Independent Control of Camera, Pose, and Expression

VDAs delegates control of camera viewpoint and upper-body pose uniformly to proxy-mesh renderings, while expressions go through the learned encoding channel. The three are independent of each other, so users can adjust expression, body pose, and viewing angle separately without interference.

In terms of training efficiency, the paper adopts a camera-triplet training strategy, completing training on a single A100 in just 34 hours using only public multi-view data. This cost is quite restrained for video diffusion model fine-tuning.

Proxy-mesh renderings refer to depth maps, normal maps, or segmentation masks rendered from a coarse parametric human/head mesh (such as SMPL-X or FLAME) at the target camera and pose, used as spatial conditioning inputs to the diffusion model. They do not aim for photorealism; they only provide a geometric scaffold: which pixels belong to the head, which direction it faces, and how deep it is.

The benefit of this approach is the separation of "control" and "generation"—control signals come from explicit geometry, precise and interpretable, while generation details are filled in by the video prior. Compared to directly embedding camera extrinsics, rendered mesh images are aligned with the image space and easier for the diffusion model's U-Net to exploit; at the same time, mesh rendering naturally supports extrapolation to arbitrary viewpoints, providing a geometric anchor for multi-view consistency.

Performance: LPIPS Reduced by 29%, JOD Improved by 66%

On the novel-view reenactment task, VDAs shows clear advantages over the current strongest single-image 3D avatar methods: on the BecomingLit dataset, facial LPIPS is reduced by 29% and the perceptual quality metric JOD improves by 66%.

The authors also conducted a static 3DGS reconstruction probe experiment, further confirming that VDAs' generated results achieve the best multi-view consistency. This point is crucial—when generating multiple views from a single image, geometric consistency across views is often the hardest link to guarantee.

LPIPS (Learned Perceptual Image Patch Similarity) uses pretrained networks such as AlexNet/VGG to extract features and compares the distance between two images in deep feature space; lower values indicate greater perceptual similarity, aligning better with human judgment than PSNR/SSIM. JOD (a perceptual distribution metric related to Jensen-Shannon Distance) measures the overall difference between generated results and ground truth at the feature distribution level, and is more sensitive to distortions such as blur and artifacts; in the paper, higher values indicate better perceptual quality (refer to the original definition for the exact direction).

The two are complementary: LPIPS emphasizes per-pixel perceptual similarity, while JOD focuses more on overall distribution quality. Reporting both metrics together avoids a single metric being "averaged out" and masking severe local distortions, such as structural collapse in the mouth region.

Physical Effects Beyond Explicit Renderers

Because it is built on a video prior, VDAs can produce effects that explicit renderers struggle to achieve: physically plausible hair dynamics, shadows, reflections, and refractions. These details are not hard-coded through physical simulation; rather, the model acquires them as implicit knowledge from video data.

This also reveals the core value of such methods: video diffusion models are not just generation tools, but carriers of world priors that can be fine-tuned into precise controllers. Once camera, pose, and expression are disentangled into independent control dimensions, this prior can be "tamed" into a practical 3D avatar system.

The paper's method details and benchmark design provide a clear reference for future research: how to preserve expressiveness without relying on 3DMM, solve view bias, and keep training costs within a reproducible range.

One-line Summary

VDAs uses video diffusion priors and disentangled expression encodings to generate multi-view consistent 3D avatars from a single image.

Paragraph Summary

The paper proposes Video Diffusion Avatars (VDAs), leveraging the appearance, motion, and physical priors learned by internet-scale video diffusion models to generate photorealistic, multi-view consistent, and controllable avatars from a single image. The method replaces the expressiveness-limited 3DMM with a learned expression space, identifies view bias in learned expression encodings, and eliminates it through disentanglement so they can serve as multi-view consistent driving signals, while building the DISTRACT benchmark to quantify this phenomenon. Camera viewpoint and body pose are controlled by proxy-mesh renderings, independent of the expression channel. A camera-triplet training strategy enables training on a single A100 in 34 hours using only public multi-view data. On the novel-view reenactment task, facial LPIPS is reduced by 29% and JOD improved by 66%, and it can also produce effects such as hair dynamics, shadows, and reflections/refractions that are difficult for explicit renderers to achieve.