Modality Gap
An empirical phenomenon in multimodal representation research whereby embeddings of different modalities trained with contrastive learning cluster separately on the unit sphere and form a stable angular separation that is difficult to eliminate through further training.
Timeline (last 90 days)
The embedding spaces of multimodal contrastive models (such as CLIP and CLAP) exhibit a modality gap: image and text embedding vectors each occupy their own region on the unit sphere and are difficult to align.
The modality gap is an empirical finding: after training, image embeddings and text embeddings cluster in two narrow cones on the unit sphere, with a stable angle between the two centroids.
Prior work has pointed out that the modality gap is related to the symmetry of the contrastive loss, the temperature coefficient, and initialization, and is difficult to eliminate by continued training.
The contribution of this paper is to point out that the modality gap is essentially the result of a geometric choice: the sphere allows modalities to separate without penalty, while the non-negativity constraint of the simplex rules out this degenerate solution.
All Facts (4)
The embedding spaces of multimodal contrastive models (such as CLIP and CLAP) exhibit a modality gap: image and text embedding vectors each occupy their own region on the unit sphere and are difficult to align.
50%UnverifiedThe modality gap is an empirical finding: after training, image embeddings and text embeddings cluster in two narrow cones on the unit sphere, with a stable angle between the two centroids.
50%UnverifiedPrior work has pointed out that the modality gap is related to the symmetry of the contrastive loss, the temperature coefficient, and initialization, and is difficult to eliminate by continued training.
50%UnverifiedThe contribution of this paper is to point out that the modality gap is essentially the result of a geometric choice: the sphere allows modalities to separate without penalty, while the non-negativity constraint of the simplex rules out this degenerate solution.
50%