Anchor Divergences: Adapting Contrastive Learning Representations to Context-Sensitive Semantic Geometry

Why Similarity Needs Context

In machine learning, measuring the similarity between two vector representations has become almost a default operation — people reach for cosine similarity almost reflexively. Yet this practice carries a hidden assumption: that semantic space has one fixed geometric structure. A new paper on arXiv (arXiv:2610.06919v1) directly challenges this assumption.

The paper's core observation is that semantic similarity is inherently context-dependent. Two images might be similar because they depict the same object, because they share a visual style, or because they relate to the same clinical diagnosis. In other words, "similarity" is never a single-dimensional concept — it shifts depending on which semantic structure we care about.

If semantics itself is multifaceted, measuring similarity across all scenarios with a single fixed cosine geometry will inevitably discard a great deal of valuable information. This is precisely the problem the authors set out to solve.

A Hidden Family of Geometries in Contrastive Learning

The paper's key insight is that representations learned through contrastive learning naturally contain a family of geometries rather than a single one. This means the same fixed set of vector representations can be specialized to different semantic structures without retraining the model.

This conclusion is compelling because it implies that even after training is complete, the representation space retains a kind of plasticity — we can dynamically redefine "what counts as similar" based on the semantic needs of a downstream task.

To make this work, the authors connect three mathematical tools: contrastive learning, exponential family distributions, and information geometry. Through the interplay among these three, the paper establishes a rigorous correspondence — probability distributions over "anchors" map precisely to Bregman geometries on the representation space.

Bregman Geometry and Anchor Distributions

Bregman divergences are a family of generalized "distance" measures that subsume Euclidean distance, KL divergence, and many other common cases; they have deep theoretical connections to exponential family distributions. The paper leverages this connection to transform the abstract problem of "choosing a semantic geometry" into the more concrete problem of "modeling a distribution over anchors."

The essence of the framework can be captured in one sentence: modeling the anchor distribution is modeling the geometry itself. Geometry is no longer an externally imposed fixed metric, but rather a learnable, specifiable object determined by data and context.

Bregman divergences were introduced by mathematician Lev Bregman in 1967. Given a strictly convex function φ, the Bregman divergence between points x and y equals φ(x) minus φ(y) minus the inner product of the gradient of φ at y with (x − y). This definition looks abstract, but its power lies in its generality: when φ is the squared norm, you recover squared Euclidean distance; when φ is the negative entropy, you get KL divergence; when φ is the negative logarithm, you obtain the Itakura-Saito divergence. The log-partition functions of exponential family distributions (Gaussian, Bernoulli, Poisson, etc.) serve exactly as the convex function φ here, creating a strict one-to-one correspondence between "which probability distribution model you choose for the anchors" and "which geometric measure you use in the representation space." Information geometry further geometrizes this relationship by endowing the parameter space with a Riemannian manifold structure, providing a theoretical foundation for switching between geometries as you switch between distribution families.

The basic paradigm of contrastive learning is: given a sample (the anchor), train the model to pull semantically similar samples (positives) closer and push unrelated samples (negatives) apart in representation space. Mainstream frameworks such as SimCLR, MoCo, and CLIP all follow this paradigm. Standard contrastive losses (e.g., InfoNCE) typically use cosine similarity as the metric, which implicitly fixes the representation space to a uniform Euclidean geometry on a hypersphere. Yet when the way positives are constructed changes — say, from data augmentation to class labels to clinical reports — the semantic structure being captured changes accordingly, while the geometric form of cosine similarity stays the same. One of the paper's key contributions is pointing out that this invariance is a limitation rather than a virtue: contrastive training actually encodes richer geometric information in the representation space than cosine similarity can express; it simply lacks a mechanism to explicitly extract and switch between these geometries.

The Anchor Divergence Method

Building on this correspondence, the paper introduces a concrete method — Anchor Divergences — that specifies the semantic geometry appropriate to a given context on top of fixed representations.

The engineering value here is worth emphasizing: the method operates on fixed representations. You do not need to retrain a large encoder for every semantic perspective. By specifying different anchor distributions over an existing representation space, you can obtain similarity measures tailored to different semantic needs. This has real implications for both computational cost and deployment flexibility.

For contexts that are highly sensitive to semantics — such as medical imaging, where the same set of images might need to be compared by anatomical structure, lesion type, or imaging modality — this ability to apply multiple geometries to a single set of representations is especially valuable.

Validation on Retrieval Tasks

The paper presents experiments on retrieval tasks, with results showing that Anchor Divergences provide an effective and efficient way to specify context-sensitive semantic similarity.

Retrieval is a classic testbed for the quality of similarity measures: given a query, the system must find the semantically closest entries in a database. If the similarity geometry can adapt flexibly to context, the same query should return different but equally valid results under different semantic perspectives. The experimental results support this, demonstrating the method's practical viability.

Standard metrics in information retrieval evaluation include Recall@K (the fraction of relevant items appearing in the top-K results), mAP (mean average precision), and NDCG (normalized discounted cumulative gain). These metrics are highly sensitive to the geometric assumptions of the similarity measure: even with identical vector representations, simply swapping the distance function can cause significant shifts in retrieval rankings. Improving retrieval performance by replacing the geometry on fixed representations therefore has ample precedent — metric learning, for instance, has long pursued learning an optimal Mahalanobis distance matrix for a given task. What distinguishes Anchor Divergences is that rather than learning a fixed linear transformation, they hand control of geometry back to the user through a probabilistic model, allowing semantic context to be specified at inference time without maintaining a separate distance matrix or retraining an adapter layer for each context.

Significance and Outlook

The theoretical contribution of this work is that it provides a unified and elegant mathematical framework for understanding the context-dependence of semantic similarity, connecting three previously somewhat separate lines of research: contrastive learning, exponential families, and information geometry.

From an application standpoint, Anchor Divergences are most appealing for their post-hoc character — they endow a representation space with context-adaptive capability without modifying the already-trained representations. As the representations produced by large multimodal models become increasingly expensive and difficult to retrain, this "lightweight specialization" approach may find practical traction in information retrieval, recommendation systems, and specialized domains such as clinical decision support.

Of course, as a newly released preprint, the method's ability to generalize to larger scales and more diverse tasks still awaits further validation, and the costs and strategies involved in modeling anchor distributions merit continued attention.