Contrastive Learning

The CLIP architecture was proposed by OpenAI in 2021 and maps images and text to the same high-dimensional vector space through contrastive learning.

The core loss functions of contrastive learning (such as InfoNCE and NT-Xent) essentially pull the representations of similar samples closer together while pushing the representations of dissimilar samples further apart.

During fine-tuning, hard negatives (e.g., positions in the same field but at a different seniority level) should be used instead of random negatives to obtain maximum gains.

Contrastive signals are more robust to thresholds and scales, and are easier to read directly from the editor's existing feature space, without having to additionally train an independent scoring network.

The typical pipeline of font embedding includes: data preparation (collecting TTF/OTF font files and rendering characters), feature extraction (encoding into vectors with a CNN or VAE), training the embedding space (contrastive learning or autoencoder reconstruction loss), dimensionality reduction visualization (t-SNE, UMAP or PCA), and observing the structure.

The Transformer architecture, contrastive learning, and reinforcement learning with human feedback (RLHF) are paradigm-level innovations.

The paper connects three mathematical tools—contrastive learning, exponential family distributions, and information geometry—to establish a rigorous mapping between probability distributions over anchor points and Bregman geometry over the representation space.

The paper proposes that representations obtained through contrastive learning inherently contain a family of geometries rather than a single geometry, and that the same set of fixed vector representations can be specialized to different semantic structures without retraining the model.

Transformer architecture and contrastive learning have been used to create embedding representations of sperm whale coda sequences.

Embedding model fine-tuning typically adopts a contrastive learning framework, training the model to bring positive sample pairs closer together and push negative sample pairs apart by constructing positive and negative sample pairs.

Geospatial foundation models typically use the Transformer architecture as their core, employing self-supervised learning methods for pre-training on large volumes of remote sensing imagery, with typical methods including masked autoencoders (MAE) and contrastive learning.

Contrastive learning naturally encourages models to capture the most discriminative semantic features in an image, rather than fine-grained spatial and quantitative information.

Contrastive learning is the core paradigm for implementing negative sample learning, which learns features by pulling the representation distance of positive pairs closer and pushing the representation distance of negative pairs farther apart.

Contrastive learning typically uses the InfoNCE loss function, which originates from the theory of Noise Contrastive Estimation.

The essential difference between JEPA and contrastive learning is that JEPA learns by predicting the representations of missing parts, without relying on explicit data augmentation strategies.

Representation collapse is a widely studied phenomenon in deep learning, and is especially common in self-supervised learning and contrastive learning frameworks.

Mainstream text embedding models are typically trained based on contrastive learning, mapping semantically similar text pairs to nearby positions in vector space and pushing unrelated text pairs further apart.

Negative samples for contrastive learning fine-tuning should use hard negatives, such as positions in the same field but at different levels, rather than random negative samples.

Fine-tuning embedding models with contrastive learning can improve matching effectiveness in recruitment scenarios, with the key being the use of hard negatives (such as positions in the same field but at a different seniority level) rather than random negatives.

Fine-tuning embedding models with contrastive learning requires labeled data (positive samples of successful hires and negative samples of rejected candidates) to achieve significant performance gains.