vid2vid-long
A video-to-video generation method that retrieves historical outputs from the input video through explicit patch-level correspondences and injects them into cross-attention, so as to maintain long-range temporal consistency in real-time streaming generation.
Timeline (last 90 days)
The vid2vid-long method comes from the paper "Consistent Video-to-Video Translation via Explicit Correspondences".
The video-to-video task naturally provides explicit cross-frame references: regions that match each other in the input video should also match each other in the output video.
For each patch of the current frame, vid2vid-long searches historical frames of the input video for matching patches to obtain patch-level correspondences.
Through correspondence cross-attention, vid2vid-long injects the retrieved historical output patches into the generation process of the current frame.
vid2vid-long's retrieval is sparse and targeted, taking only the patches that the current frame actually needs, so the computational overhead is controllable.
When vid2vid-long is plugged into multiple recent video models, long-range temporal consistency improves consistently across all of them.
After introducing vid2vid-long's retrieval mechanism, generation quality does not degrade.
vid2vid-long maintains real-time performance, trading no latency for consistency.
The effectiveness of vid2vid-long depends on the reliability of patch matching on the input side; when the input video contains occlusions, violent motion, or large illumination changes, the accuracy of the correspondences is affected.
Optical flow typically estimates only pixel-level motion between adjacent frames, whereas vid2vid-long's correspondences are patch-level semantic matches across long-distance frames.
1 more timeline events
All Facts (11)
The vid2vid-long method comes from the paper "Consistent Video-to-Video Translation via Explicit Correspondences".
50%UnverifiedThe video-to-video task naturally provides explicit cross-frame references: regions that match each other in the input video should also match each other in the output video.
50%UnverifiedFor each patch of the current frame, vid2vid-long searches historical frames of the input video for matching patches to obtain patch-level correspondences.
50%UnverifiedThrough correspondence cross-attention, vid2vid-long injects the retrieved historical output patches into the generation process of the current frame.
50%Unverifiedvid2vid-long's retrieval is sparse and targeted, taking only the patches that the current frame actually needs, so the computational overhead is controllable.
50%UnverifiedWhen vid2vid-long is plugged into multiple recent video models, long-range temporal consistency improves consistently across all of them.
50%UnverifiedAfter introducing vid2vid-long's retrieval mechanism, generation quality does not degrade.
50%Unverifiedvid2vid-long maintains real-time performance, trading no latency for consistency.
50%UnverifiedThe effectiveness of vid2vid-long depends on the reliability of patch matching on the input side; when the input video contains occlusions, violent motion, or large illumination changes, the accuracy of the correspondences is affected.
50%UnverifiedOptical flow typically estimates only pixel-level motion between adjacent frames, whereas vid2vid-long's correspondences are patch-level semantic matches across long-distance frames.
50%Unverifiedvid2vid-long represents the trend of turning the "memory" problem in video generation into a "retrieval" problem.
50%