Gen4P
A model framework that repurposes a pretrained video generator as a feed-forward 4D perceiver. Through an on-demand query mechanism, it skips iterative denoising and directly recovers 4D geometry and motion information from monocular video.
Timeline (last 90 days)
Gen4P proposes that a pretrained video generator can be repurposed into a feed-forward 4D perceiver that recovers 4D geometry and motion information from monocular video.
Gen4P demonstrates that a pretrained video generator can directly serve as a 4D perceiver without iterative denoising.
Gen4P builds on an "on-demand querying" mechanism, letting clean video latents pass through a single encoder forward pass and then predicting 4D attributes through a lightweight cross-attention decoder.
The 4D attributes predicted by Gen4P include 3D point trajectories, video depth and camera poses.
Gen4P's continuous-time queries allow the model to query 4D information at arbitrary spatiotemporal locations, rather than being limited to a fixed grid or frame rate.
The research team introduces a pretrained video depth LoRA into Gen4P as a frozen geometric prior, improving overall performance without adding extra trainable parameters.
Gen4P achieves state-of-the-art or competitive performance on 3D point tracking, video depth estimation and camera pose estimation tasks.
Qualitative results on in-the-wild videos show that Gen4P can track complex motion and preserve fine structures, with a degree of robustness in the face of occlusion, fast motion and complex scenes.
The project page of Gen4P is https://gen4p.github.io.
All Facts (9)
Gen4P proposes that a pretrained video generator can be repurposed into a feed-forward 4D perceiver that recovers 4D geometry and motion information from monocular video.
50%UnverifiedGen4P demonstrates that a pretrained video generator can directly serve as a 4D perceiver without iterative denoising.
50%UnverifiedGen4P builds on an "on-demand querying" mechanism, letting clean video latents pass through a single encoder forward pass and then predicting 4D attributes through a lightweight cross-attention decoder.
50%UnverifiedThe 4D attributes predicted by Gen4P include 3D point trajectories, video depth and camera poses.
50%UnverifiedGen4P's continuous-time queries allow the model to query 4D information at arbitrary spatiotemporal locations, rather than being limited to a fixed grid or frame rate.
50%UnverifiedThe research team introduces a pretrained video depth LoRA into Gen4P as a frozen geometric prior, improving overall performance without adding extra trainable parameters.
50%UnverifiedGen4P achieves state-of-the-art or competitive performance on 3D point tracking, video depth estimation and camera pose estimation tasks.
50%UnverifiedQualitative results on in-the-wild videos show that Gen4P can track complex motion and preserve fine structures, with a degree of robustness in the face of occlusion, fast motion and complex scenes.
50%UnverifiedThe project page of Gen4P is https://gen4p.github.io.
50%