Cross-Modal Misalignment
A failure mode of multimodal models in which audio information is incorrectly bound to visual information, such as mismatching speaker identity or sound source location. It exposes deficiencies in audio-visual alignment mechanisms.
Timeline (last 90 days)
OmniCapBench can diagnose four typical failure modes: temporal localization failure, identity drift, cross-modal misalignment, and hallucinated description.
Identity drift and cross-modal misalignment are the two most prominent weak points of current frontier MLLMs in audio-visual captioning tasks.
Cross-modal misalignment refers to the model incorrectly binding audio information to visual information, for example A is speaking in the picture but it is B who is vocalizing in the audio, yet the model describes A as speaking.
All Facts (3)
OmniCapBench can diagnose four typical failure modes: temporal localization failure, identity drift, cross-modal misalignment, and hallucinated description.
50%UnverifiedIdentity drift and cross-modal misalignment are the two most prominent weak points of current frontier MLLMs in audio-visual captioning tasks.
50%UnverifiedCross-modal misalignment refers to the model incorrectly binding audio information to visual information, for example A is speaking in the picture but it is B who is vocalizing in the audio, yet the model describes A as speaking.
50%