NVIDIA's New Research: Joint Video and 3D Motion Generation, Any Rig Can Move

Video generation models can already depict fairly realistic motion—monkeys leaping over bars, robots walking through corridors, dancers spinning and jumping. But a fundamental problem remains unresolved: these motions exist only in pixels. You cannot extract a jump from a video, bind it to a specific 3D model, and play it back in a 3D scene. NVIDIA PAR Lab's latest research attempts to break down this barrier.

Beyond Pixels: Why Video Motion Is Hard to Reuse

Current video generation models are impressive enough in motion expressiveness, but their output is essentially a 2D image sequence. If you want to transfer motion from a video to 3D assets, traditional approaches either rely on motion capture pipelines specific to human skeletons or triangulate 3D motion from generated 2D pose maps. The former has limited generalization ability, while the latter is very fragile in the face of occlusion and depth ambiguity.

The NVIDIA team presents a more direct solution in the paper: let the model generate video and 3D motion simultaneously, work for any given rig, and output a camera parameter to register the motion into the scene. In other words, the model not only tells you "what happened" but also "where in 3D space it happened."

What is a "rig": In 3D animation and graphics, a rig refers to a skeletal hierarchy bound to a 3D model, including joints and their connections, rotation axes, and skinning weights. Animators drive the model's motion by rotating and moving these joints, like manipulating a puppet's skeleton. Different characters (humans, robots, animals) have rigs that differ greatly in joint count, topology, and naming—this is exactly why "any rig" becomes a technical challenge: the model cannot assume a fixed skeleton template but must learn to map motion from text or video onto any given set of joints.

Decoupled Architecture: Letting the Motion Model Focus on Learning Motion

The core design choice of this work is decoupling 3D motion generation from video generation. According to the paper, some previous joint video-motion models only support human skeletons yet add billions of parameters for this; NVIDIA's approach keeps 3D motion in a separate model that supports arbitrary skeleton structures, then connects it to a pretrained video model through a small number of cross-attention layers.

This separation brings a practical benefit: the motion model can first learn how a skeleton moves using pure motion data, which is easier to obtain and augment than paired video-motion data. The video model handles what it excels at—visual synthesis. The two are bridged via cross-attention rather than forcibly merged into one massive network.

Joint Localization and Camera Recovery

The model also learns to localize each joint's position in video frames. This capability seems auxiliary but is actually key to putting 3D motion back into the scene—with joints' positions in the image, one can infer camera parameters and thus complete registration of the 3D motion with the scene. The paper emphasizes that this makes the generated motion "respond to the surrounding scene" rather than being an isolated motion floating in a vacuum.

Cross-attention: Cross-attention is a mechanism in Transformers that allows one sequence (e.g., intermediate features of a video model) to "query" information from another sequence (e.g., hidden states of a 3D motion model). In this architecture, features from the video branch serve as query, and features from the motion branch serve as key and value, thereby passing information between the two so that video generation is aware of the ongoing 3D motion, and vice versa. Compared with merging two models into one large network, inserting only a few cross-attention layers has low parameter overhead, more stable training, and preserves the capabilities of each pretrained model.

Camera parameters and scene registration: To recover 3D information from a 2D image, one must know the camera intrinsics (focal length, principal point) and extrinsics (rotation, translation) used to capture that image. The model predicts each joint's pixel position in the image while generating video, effectively providing 2D-3D correspondences, which can then be used to solve for camera parameters via geometric methods (such as the PnP algorithm). With camera parameters, the generated motion can be aligned with the 3D coordinate system of the video background, ensuring that when played back in a game engine or simulation environment, the relative position of the motion and the scene remains consistent rather than floating arbitrarily.

Building Training Data with a Generative Video Editor

The difficulty in training a model that supports multiple skeletons lies in data. Human videos are easy to find, but paired video-motion data for robots and various non-human skeletons is scarce. The research team's approach is quite clever: using a generative video editor, they place animated 3D assets into scenes that match their motion, and filter the edited results to ensure the known motion remains valid.

Combining human and robot videos, this pipeline ultimately produced nearly 200,000 training pairs. This number is considerable in the field of joint video-motion generation and explains why the model can cover diverse skeleton types.

Empirical Performance: Multi-skeleton Animation More Faithful to Text

Evaluation results show that the model's fidelity in animating diverse skeletons from text outperforms "video-first" pipelines, and the generated motion also responds to the surrounding scene. More noteworthy is its versatility—the same model can perform conversions in multiple directions:

• Video to skeleton motion: Extract replayable 3D motion from video
• Skeleton motion to video: Generate corresponding visual frames given motion
• Keyframe-controlled 3D motion: Manipulate generated motion via keyframes

This bidirectional capability means it is not just a generation tool but can also serve as a conversion layer in 3D animation workflows. For creators who need to turn video footage into editable 3D animation, this direction is worth continued attention.

Potential Impact on 3D Content Creation Pipelines

This research addresses the gap between video generation and 3D asset pipelines. Currently, content produced by video models often requires extensive manual retargeting and cleanup before entering game engines, film previsualization, or robot simulation. A model that can directly output replayable 3D motion and complete scene registration could theoretically reduce friction in this chain.

Of course, the paper currently presents research-stage results, and performance when generalizing to complex scenes, multi-object interactions, and long-horizon motions remains to be further validated. However, the architectural idea of "separating the motion model from the video model and bridging with cross-attention" has reference value for subsequent research on arbitrary-rig motion generation. The project page has released videos and supplementary results, and interested readers can directly view the actual results.

One-line Summary

NVIDIA's JCA model decouples motion and video generation, supporting arbitrary rigs and outputting replayable 3D motion.

Paragraph Summary

NVIDIA PAR Lab proposes the JCA model to break down the barrier between video generation and 3D motion: the model can simultaneously generate video and replayable 3D motion from text, and output camera parameters to register the motion into the scene. Its core design decouples the 3D motion model from a pretrained video model, bridging them with a small number of cross-attention layers to support arbitrary skeleton structures, avoiding billions of parameters added for human-specific skeletons. For training data, the team uses a generative video editor to synthesize paired samples, combining human and robot videos to build nearly 200,000 pairs. Evaluations show that the model is more faithful than video-first pipelines in text-driven animation of multiple skeletons and supports three directions: video-to-skeleton motion, skeleton motion-to-video, and keyframe-controlled 3D motion.