InfoNCE
A loss function commonly used in self-supervised contrastive learning. It learns high-quality representations by maximizing the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, and is widely applied in vision, audio, and recommendation system domains.
Timeline (last 90 days)
Most embedding models are trained using contrastive loss (InfoNCE or its variants), which only cares about the relative ranking of positive samples and not the absolute value of scores.
Adding the same constant to every similarity score in the softmax does not change the loss, so the model can arbitrarily choose the score range for unrelated texts, and the training process pays no attention to absolute scores whatsoever.
Mainstream contrastive learning frameworks such as SimCLR, MoCo, and CLIP typically use cosine similarity as the metric, implicitly fixing the representation space to a uniform Euclidean geometry on a hypersphere.
Sona's tokenizer is trained with an InfoNCE contrastive loss using two types of positive sample pairing signals: tracks co-occurring within the same recommendation response from different artists, and tracks by the same artist with high co-listening NPMI within a three-month window.
Contrastive learning typically uses the InfoNCE loss function, which originates from the theory of Noise Contrastive Estimation.
All Facts (5)
Most embedding models are trained using contrastive loss (InfoNCE or its variants), which only cares about the relative ranking of positive samples and not the absolute value of scores.
50%UnverifiedAdding the same constant to every similarity score in the softmax does not change the loss, so the model can arbitrarily choose the score range for unrelated texts, and the training process pays no attention to absolute scores whatsoever.
50%UnverifiedMainstream contrastive learning frameworks such as SimCLR, MoCo, and CLIP typically use cosine similarity as the metric, implicitly fixing the representation space to a uniform Euclidean geometry on a hypersphere.
50%UnverifiedSona's tokenizer is trained with an InfoNCE contrastive loss using two types of positive sample pairing signals: tracks co-occurring within the same recommendation response from different artists, and tracks by the same artist with high co-listening NPMI within a three-month window.
50%UnverifiedContrastive learning typically uses the InfoNCE loss function, which originates from the theory of Noise Contrastive Estimation.
50%