Simplex-Constrained Representation: 97% Elimination of the Multimodal Embedding Modality Gap

Multimodal contrastive models (such as CLIP and CLAP) can learn shared representations, but their embedding space suffers from a stubborn modality gap—image and text embeddings each occupy their own region on the unit sphere and are difficult to truly align. A new paper proposes a purely geometric solution: replace the unit sphere with a probability simplex as the embedding domain, forcing both modalities into the same coordinate system through non-negativity and unit-mass constraints. Experiments show that this method reduces the ℓ2 distance between centroids by 97–99% on five multimodal retrieval benchmarks, while retrieval performance matches or even exceeds that of unconstrained models.

The Root of the Modality Gap and a New Geometric Solution

CLIP-style models map images and text into the same embedding space through contrastive learning, but research shows that the embeddings of the two modalities often occupy opposite sides of the unit sphere, forming the so-called "modality gap." This separation is not caused by semantic differences but is a byproduct of embedding geometry: the symmetry of the unit sphere S^{d-1} allows the model to push different modalities into different regions without affecting the contrastive loss.

The authors point out that although the unit sphere and the probability simplex Δ^{d-1} have the same intrinsic dimension, the simplex's non-negativity constraint and unit-mass constraint constitute a structural prior. This prior acts like a cross-modal calibrator, forcing the representations of both modalities into the positive orthant of the same coordinate system, thereby naturally suppressing separation.

The unit sphere and the probability simplex are key to understanding the geometric approach of this paper. The unit sphere $S^{d-1} = {x \in \mathbb{R}^d : |x|_2 = 1}$ is the most commonly used constraint surface in embedding learning, and cosine similarity is naturally defined on it; its symmetry is extremely high, with all directions being equivalent, so the model can place different modalities of the same semantics at two distant positions on the sphere without increasing the contrastive loss. The probability simplex $\Delta^{d-1} = {p \in \mathbb{R}^d : p_i \ge 0, \sum_i p_i = 1}$ is the geometric representation of probability distributions. It also has intrinsic dimension $d-1$, but it occupies only a small region in the positive orthant of the sphere, and its boundary is jointly defined by non-negativity and translation constraints. The key difference between the two lies not in dimension but in the shape of the constraint: the simplex restricts all representations to the positive orthant of the same coordinate system, eliminating degenerate solutions such as "opposite regions" on the sphere, and thus provides a structural prior for cross-modal alignment.

Lightweight Adapter and Total Variation Similarity

In implementation, the authors append a lightweight softmax adapter after the original embedding to map vectors on the unit sphere to the probability simplex. This adapter has very few parameters and does not significantly change the model size. At the same time, the similarity measure is changed from the commonly used cosine similarity to Total Variation similarity, which is better suited to comparing probability distributions on the simplex.

The computational overhead of this design is almost negligible, yet it directly affects the geometric structure of the embeddings. The authors validated the effect on five multimodal retrieval benchmarks: the ℓ2 distance between embedding centroids decreased by 97–99%, meaning that the two modalities almost coincide in space.

Total Variation similarity originates from the total variation distance between probability distributions. For discrete distributions $p, q$, it is defined as $TV(p,q) = \frac{1}{2}\sum_i |p_i - q_i|$, taking values between 0 and 1, equivalent to $1 - \sum_i \min(p_i, q_i)$. Unlike cosine similarity, which only cares about the angle between vectors, total variation directly compares the mass overlap of two distributions in each dimension, which is especially natural for points on the simplex: it measures "how much probability mass is shared." In retrieval tasks, this means that similarity scores have a clear probabilistic interpretation and are more sensitive to sparsity and tail mass changes than inner products. The actual computation only requires summing the absolute differences element by element, with complexity on the same order as cosine similarity, so the adaptation cost is very low.

Empirical Evidence That Performance Improves Rather Than Degrades

Geometric alignment does not sacrifice retrieval performance. The paper reports that the retrieval performance of the simplex-constrained model matches or exceeds that of the standard unconstrained model. This suggests that the narrowing of the modality gap is not achieved by losing semantic information, but by unlocking the representational potential through a more reasonable geometric structure.

The advantage of this method is that it is purely geometric and model-agnostic: it does not depend on a specific contrastive loss or network architecture and can be plug-and-play applied to existing CLIP/CLAP-style models. For applications requiring cross-modal retrieval (such as image-text search and audio-text matching), this means a more compact embedding space and potentially more robust generalization.

The modality gap is an empirical finding in recent multimodal representation research: even though the contrastive learning objective only requires matched sample pairs to be close and unmatched pairs to be far apart, after training the image embeddings and text embeddings still cluster in two narrow cones on the unit sphere, with a stable angle between the two centroids. Prior work has pointed out that this phenomenon is related to the symmetry of the contrastive loss, the temperature coefficient, and initialization, and is difficult to eliminate by continued training. The contribution of this paper is to point out that the gap is essentially the result of a geometric choice—the sphere allows modality separation without penalty, while the simplex's non-negativity constraint rules out this degenerate solution. Therefore, narrowing the gap does not require additional loss terms or post-hoc alignment; it only requires changing the constraint surface on which the embeddings live.

Limitations and Outlook

The paper currently validates only on retrieval tasks and has not yet addressed generation tasks or zero-shot classification. Whether the simplex constraint limits the expressive freedom of embeddings, and the training stability under total variation similarity, still require broader experiments. However, this approach provides a new perspective for multimodal representation learning: rather than chasing on the sphere, change the geometric space so that alignment happens naturally.

One-line Summary

Change embeddings from the unit sphere to the probability simplex, using geometric constraints to eliminate the modality gap in multimodal embeddings.

Paragraph Summary

The article introduces a new purely geometric approach to multimodal alignment: although contrastive models such as CLIP and CLAP map images and text into the same unit-sphere space, the embeddings of the two modalities often occupy opposite sides of the sphere, forming a stubborn "modality gap"—not a semantic difference but a byproduct of spherical symmetry. The paper proposes changing the embedding domain from the unit sphere to the probability simplex, using non-negativity and unit-mass constraints as a structural prior to force both modalities into the same coordinate system; in implementation, it only requires appending a lightweight softmax adapter after the embedding and replacing cosine similarity with total variation similarity, which is better suited to probability distributions, with almost negligible computational overhead. On five multimodal retrieval benchmarks, the ℓ2 distance between the centroids of the two modalities' embeddings decreases by 97–99%, and retrieval performance matches or exceeds that of unconstrained models. The method is purely geometric and model-agnostic, and can be plug-and-play applied to existing models, but it has so far been validated only on retrieval tasks; generation and zero-shot classification remain to be explored.