SyncNet
An audio-visual synchronization detection model based on a two-stream convolutional network, used to determine whether facial movement and speech in a video are synchronized, thereby localizing the active speaker.
Timeline (last 90 days)
The VoxCeleb 1 dataset was built through an automated pipeline from the VGG Face dataset using a HOG-based face detector, SyncNet, and a VGG Face CNN, so bias in the face recognition domain propagates directly into the speaker verification dataset.
Common metrics for evaluating lip-sync quality include LSE (Lip Sync Error) and SyncNet-based confidence scores.
The author considers the lip-sync score only a coarse filter, noting that the 12-frame sample is too small and LLM scoring is noisy, and that a more reliable approach would use MediaPipe keypoint measurements or SyncNet scoring.
SyncNet outputs a lip-sync score by comparing mouth visual features in video frames with audio MFCC features, and is widely used in academic lip-sync benchmarks; MediaPipe's FaceMesh module can track 468 facial keypoints per frame.
All Facts (4)
The VoxCeleb 1 dataset was built through an automated pipeline from the VGG Face dataset using a HOG-based face detector, SyncNet, and a VGG Face CNN, so bias in the face recognition domain propagates directly into the speaker verification dataset.
50%UnverifiedCommon metrics for evaluating lip-sync quality include LSE (Lip Sync Error) and SyncNet-based confidence scores.
50%UnverifiedThe author considers the lip-sync score only a coarse filter, noting that the 12-frame sample is too small and LLM scoring is noisy, and that a more reliable approach would use MediaPipe keypoint measurements or SyncNet scoring.
50%UnverifiedSyncNet outputs a lip-sync score by comparing mouth visual features in video frames with audio MFCC features, and is widely used in academic lip-sync benchmarks; MediaPipe's FaceMesh module can track 468 facial keypoints per frame.
50%