Gen4P: How a Video Generator Becomes a 4D Perceiver

The potential of video generation models may extend far beyond producing visually appealing footage. A new study called Gen4P proposes that a pretrained video generator can be repurposed into a feed-forward 4D perceiver that recovers 4D geometry and motion from monocular video. This work opens a new path for downstream applications of video generation pretraining.

Core Idea: Skip Iterative Denoising, Perceive 4D Directly

Traditionally, using diffusion models for perception tasks often requires an iterative denoising process, which is computationally expensive. Gen4P's breakthrough lies in demonstrating that a pretrained video generator can act directly as a 4D perceiver without iterative denoising.

Specifically, Gen4P builds on an "on-demand querying" mechanism. It passes clean video latents through a single encoder forward pass, then uses a lightweight cross-attention decoder to predict the required 4D attributes—such as 3D point trajectories, video depth, and camera pose. This design shifts the generative model from "drawing it out step by step" to "reading it out in one shot," greatly simplifying the inference pipeline.

What are video latents? In diffusion-based video generation models, raw video pixels are typically compressed by a pretrained autoencoder (such as a VAE) into a low-dimensional, low-resolution latent representation. This compressed representation is called "video latents." It can be understood as a compact encoding of the video inside the model, preserving key information about visual content and motion while greatly reducing computation. The diffusion model's denoising process usually happens in this latent space rather than directly in pixel space. Gen4P's "clean video latents" refer to encoded latent representations that contain no noise and no longer need step-by-step denoising before the decoder can read them. Understanding this concept helps clarify why Gen4P can skip iterative denoising: it bypasses the diffusion sampling process, performs a single forward pass in latent space, and then reads out 4D attributes through a decoder, thereby turning the generative model's "latent-space knowledge" into perception capability.

Continuous-Time Queries and Geometric Priors

In terms of flexibility, Gen4P's on-demand querying formulation supports flexible spatial sampling, while its proposed continuous-time queries further provide flexibility along the temporal dimension. This means the model can query 4D information at arbitrary spatiotemporal locations, rather than being constrained to a fixed grid or frame rate.

To enhance geometric knowledge, the research team also proposed a straightforward plug-in approach: introducing a pretrained video depth LoRA as a frozen geometry prior. This improves overall performance without adding extra trainable parameters. For researchers who want to quickly adapt to downstream tasks, this way of injecting domain knowledge without retraining is quite appealing.

LoRA and Frozen Geometry Priors LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Its core idea is to insert a pair of low-rank matrices alongside the weights of a pretrained model, training only these small newly added parameters while keeping the original model weights frozen. The benefits are low GPU memory and storage overhead, as well as fast adaptation to new tasks. The "video depth LoRA" mentioned by Gen4P refers to a module that has already been fine-tuned with LoRA on a video depth estimation task, and it has encoded rich monocular depth and geometric cues. The research team connects this module to the model as a "frozen geometry prior," meaning that none of its parameters are updated during use; instead, its learned geometric knowledge is leveraged to enhance 4D prediction. Because it is plug-in style, researchers do not need to retrain the entire model and add almost no trainable parameters, yet can inject depth priors into a task originally focused on generation. This approach is in the same spirit as "prompt engineering" or "adapters," offering a practical path for quickly reusing existing model capabilities.

Performance: Leading or Competitive Across Multiple Tasks

Gen4P achieves state-of-the-art or competitive performance on multiple tasks, including 3D point tracking, video depth estimation, and camera pose estimation. These tasks cover the core capabilities of 4D perception.

Qualitative results on in-the-wild videos are especially noteworthy: Gen4P can track complex motion while preserving fine structure. This shows that the model performs well not only on standard benchmarks, but also has a degree of robustness when facing occlusion, fast motion, and complex scenes common in real-world video.

3D point tracking, depth estimation, and camera pose estimation These three tasks are core components of 4D perception. 3D point tracking aims, given several query points in a certain frame of a video, to predict the 3D coordinate trajectories of these points in all subsequent frames. The difficulty lies in handling occlusion, deformation, and fast motion. Video depth estimation requires generating a per-pixel depth map for each frame of a video while maintaining inter-frame depth consistency to avoid flickering. Camera pose estimation infers the position and orientation of the recording camera in each frame from video, and is often related to structure from motion (SfM) and SLAM problems. The three are interrelated: accurate depth helps recover 3D structure, stable pose helps align trajectories, and 3D trajectories in turn can constrain depth and pose. Gen4P can handle all three types of tasks within a unified framework and achieve leading or competitive results, indicating that the 4D representation it learns has strong generality.

Significance and Outlook

Gen4P's core contribution is demonstrating a possibility: the spatiotemporal understanding accumulated through video generation pretraining can be transferred to unified 4D perception tasks. This offers inspiration in two directions for future research—first, how to more efficiently use generative models for discriminative tasks; second, how to enable large models to quickly acquire specific capabilities through lightweight adaptation (such as LoRA priors).

Currently, more visual results can be viewed on the project page https://gen4p.github.io. For researchers and developers interested in the intersection of video understanding, 3D reconstruction, and generative models, Gen4P is a direction worth following.

What is feed-forward perception? "Feed-forward" means that a model can produce a prediction through only one or a few forward passes, without requiring the hundreds or thousands of iterative sampling steps used by diffusion models. In traditional diffusion-based perception methods, to recover a clean result from noise, the denoising network often must be called repeatedly, and inference time grows linearly with the number of steps, making it difficult to meet real-time or large-scale application needs. Gen4P transforms perception tasks into a feed-forward process of one encoding plus one decoding, significantly reducing computational latency and bringing video generation pretrained models closer to becoming practical perception tools. The significance of this paradigm shift is that generative models are no longer just content creators; they can also become efficient structure understanding systems. Of course, feed-forward methods also face challenges, such as how to preserve enough detail and ambiguity in a single pass, which remains a direction for future research.

One-line Summary

Gen4P enables a pretrained video generator to recover 4D geometry and motion in a feed-forward way without iterative denoising.

Paragraph Summary

Gen4P proposes reusing a pretrained video generator as a feed-forward 4D perceiver to recover 3D point trajectories, video depth, and camera pose from monocular video. Its core is an "on-demand querying" mechanism: clean video latents pass through a single encoder forward pass, then a lightweight cross-attention decoder predicts 4D attributes, thereby skipping the iterative denoising of traditional diffusion models. The method also introduces continuous-time queries, supports sampling at arbitrary spatiotemporal locations, and uses a frozen video depth LoRA as a geometry prior to improve performance without adding trainable parameters. Experiments show that Gen4P achieves leading or competitive performance in 3D point tracking, depth estimation, and camera pose estimation, and can track complex motion while preserving detail in in-the-wild videos. This work demonstrates that the spatiotemporal understanding accumulated through video generation pretraining can be transferred to unified 4D perception tasks, offering new ideas for efficiently using generative models for discriminative tasks and lightweight adaptation.