CLIP vs SigLIP: Core Differences in Vision Encoder Training Paradigms

Before a vision-language model (VLM) can perform tasks like image classification or video question answering, input images or videos must first be converted into representations the model can "understand."

Vision-Language Models (VLMs) represent a core technical direction in today's multimodal AI landscape, embodying machines' ability to simultaneously understand visual and linguistic information. Unlike traditional single-modality models, VLMs can process visual inputs such as images and videos while combining textual information for reasoning. Typical applications include image captioning, Visual Question Answering (VQA), and image-text retrieval. In recent years, breakthroughs with models like CLIP, Flamingo, and LLaVA have demonstrated VLMs' powerful zero-shot transfer capabilities and cross-modal understanding, making them a crucial research direction toward Artificial General Intelligence (AGI). The core challenge for VLMs lies in effectively mapping visual information to the semantic space of language models—precisely the key problem vision encoders aim to solve.

This critical task falls to the vision encoder. Interestingly, despite contemporary mainstream vision encoders sharing highly similar underlying architectures (nearly all Transformer-based), their true differences lie not in model structure but in their training objectives. This article provides an in-depth comparison of two representative vision encoder training paradigms: CLIP and SigLIP.

Vision Encoders: The "Eyes" of VLMs

The core responsibility of a vision encoder is to transform images into numerical representations that VLMs can process. Currently, the vast majority of vision encoders are built on the Transformer architecture: the model divides an image into multiple patches, then converts each patch into vectorized visual embeddings.

Transformer was originally proposed by Google in 2017 for natural language processing, with self-attention mechanisms at its core. In 2020, the emergence of Vision Transformer (ViT) marked Transformer's successful entry into computer vision. ViT divides images into fixed-size patches (typically 16×16 or 14×14 pixels), flattens each patch, and converts them into embedding vectors through linear projection. These vector sequences are then fed into Transformer encoders for processing. Compared to traditional Convolutional Neural Networks (CNNs), ViT can model global dependencies while avoiding CNN's receptive field limitations. This architecture demonstrates exceptional performance after large-scale pretraining and has become the mainstream choice for current vision encoders. Notably, ViT requires substantial data to achieve optimal results, which has driven the development of efficient pretraining methods like contrastive learning.

After obtaining these embeddings, many VLMs pass them through a projector—typically a linear layer or MLP—to map the image embedding dimensions to the dimensional space expected by the large language model (LLM), thus completing the connection between visual information and language models.

Since most vision encoders share the same model design, what's the key factor that truly differentiates them? The answer lies not in architecture but in how they are trained. This is the starting point for understanding the differences between CLIP and SigLIP.

CLIP: The Classic Contrastive Learning Paradigm

CLIP stands for Contrastive Language-Image Pre-training. It learns to understand images through image-text pairings. Its core objective is to "pull together" matching image-text pairs while "pushing apart" mismatched ones.

Contrastive learning is a self-supervised learning paradigm with the core idea of "pulling similar samples together and pushing dissimilar ones apart." Mathematically, contrastive learning typically employs the InfoNCE loss function, derived from Noise Contrastive Estimation theory. Given a positive sample pair and multiple negative samples, the model must maximize similarity for the positive pair while minimizing similarity with negative samples. This method requires no manual annotation and can learn meaningful representations from large-scale unlabeled data. In computer vision, methods like SimCLR and MoCo generate positive and negative sample pairs through data augmentation, while in the multimodal domain, CLIP leverages natural image-text pairing relationships, avoiding complex sample construction processes. The key to contrastive learning's success lies in the quality and quantity of negative samples—more diverse negative samples help models learn finer discrimination capabilities.

Dual-Tower Architecture and Similarity Matrix

CLIP relies on a typical "two-tower" system: a pretrained vision encoder (like ViT) and a pretrained text encoder. Images pass through ViT to produce corresponding vector embeddings, while text descriptions pass through the text encoder to generate corresponding text embeddings.

Based on these pairings, the model constructs a similarity matrix, comparing each image embedding with every text embedding pairwise. Each cell in the matrix stores the cosine similarity between an image-text pair.

Cosine similarity measures the directional similarity of two vectors in high-dimensional vector space, calculated as cos(θ) = (A·B)/(||A||×||B||), with values ranging from [-1,1]. In deep learning embedding spaces, cosine similarity is more suitable than Euclidean distance for measuring semantic similarity because it focuses on vector direction rather than magnitude. When two embedding vectors point in the same direction (cosine similarity near 1), they are highly semantically related; perpendicular directions (near 0) indicate irrelevance; opposite directions (near -1) indicate semantic opposition. In models like CLIP, by learning to map semantically similar image-text pairs to nearby directions in embedding space, models achieve cross-modal semantic alignment. Note that vectors typically require L2 normalization before computing cosine similarity, making dot product operations equivalent to cosine similarity calculations, simplifying implementation.

Contrastive Learning Mechanisms and Limitations

CLIP then applies contrastive learning across the entire matrix. At a high level, this contrastive learning resembles computing categorical cross-entropy along the rows and columns of the matrix. Through the softmax function, the model attempts to assign the highest probabilities to matching "image-text" and "text-image" pairs.

CLIP's power lies in transforming learning from simple label mapping to deeper semantic understanding, enabling zero-shot classification—recognizing categories never explicitly seen during training.

Zero-shot learning refers to a model's ability to recognize or process categories never seen during training—an important hallmark of artificial general intelligence. Traditional supervised learning models can only work on predefined category sets and must be retrained for new categories. Models like CLIP, by learning joint embedding spaces for images and text, can understand arbitrary new categories described in natural language. For example, even if training data contains no images of "snow leopards," as long as the model has seen related concepts like "leopard" and "snow," it can recognize snow leopard images through the text prompt "a photo of a snow leopard." This capability stems from language's compositionality and generalization—natural language can describe infinite conceptual combinations, and vision encoders inherit this generalization ability through alignment with language. Zero-shot learning greatly reduces AI system deployment costs, enabling models to quickly adapt to new scenarios without additional annotation.

However, CLIP introduces a specific structural problem: samples compete with each other within training batches. Imagine if multiple text descriptions in a batch all reasonably match a given image—what happens? Due to softmax's global normalization mechanism, these samples that should all be "correct" are forced into mutual exclusion, potentially harming learning effectiveness.

The softmax function is the most commonly used multi-class activation function in neural networks, converting arbitrary real-valued vectors into probability distributions. Its mathematical form is softmax(xi) = exp(xi)/Σexp(xj), ensuring all outputs sum to 1 and are non-negative. In CLIP, softmax operates on rows or columns of the similarity matrix, forcing the model to select the best match among all candidates. However, this global normalization introduces an inherent contradiction: when multiple reasonable positive sample pairs exist in a batch (e.g., different text descriptions of the same image), softmax's normalization constraint artificially reduces these positive samples' probabilities since their sum must equal 1. This "winner-takes-all" competition mechanism may lead models to learn suboptimal representations. Additionally, softmax requires computing the normalization term across the entire batch, introducing cross-device communication overhead in distributed training and limiting scalability. These limitations prompted researchers to explore alternatives, with SigLIP's sigmoid loss being one such outcome.

SigLIP: Redefining Training Objectives with Sigmoid

SigLIP stands for Sigmoid Loss for Language-Image Pre-training. It retains many of CLIP's core features: it also has image and text encoders, generates embeddings for images and text, and maps similarity scores into a similarity matrix.

The true divergence lies in loss function design.

From "Global Competition" to "Independent Judgment"

CLIP learns image-text similarity by applying softmax across the entire batch, causing potential matches to compete with each other. SigLIP abandons this global normalization, instead treating each image-text pair as an independent binary prediction.

By applying the sigmoid function to each pair's score, the model estimates whether this image matches this text. In other words, the learning objective undergoes a fundamental transformation:

• CLIP's question: Among these candidates, which text describes this image?
• SigLIP's question: Is this specific "image-text" pairing a valid match—yes or no?

The sigmoid function σ(x)=1/(1+e^(-x)) is a classic activation function in neural networks, mapping real numbers to the (0,1) interval, commonly used for binary classification tasks. Unlike softmax's multi-class mutual exclusivity, sigmoid treats each prediction as an independent binary judgment—"yes" or "no." In SigLIP, each image-text pair's similarity score, after passing through sigmoid, is interpreted as the probability that the pair matches, independent of other samples in the batch. This design allows multiple positive samples to simultaneously receive high probabilities, avoiding the mutual suppression problem in CLIP. From a loss function perspective, sigmoid paired with binary cross-entropy can independently optimize each sample pair without requiring global normalization. This locality makes sigmoid loss naturally suited for parallel computation—each device can independently process its subset of samples without frequent cross-device synchronization, crucial for large-scale distributed training.

This shift in perspective seems subtle but fundamentally redefines the optimization task's nature. Each sample is no longer forced to compete with other samples in the batch but is judged independently.

More Efficient Distributed Scalability

Since SigLIP no longer requires the softmax normalization that CLIP depends on, its training objective can scale more efficiently on large-scale distributed systems. It removes the hard constraint that "every sample must participate in the same shared normalization operation," making batch size expansion and cross-device parallel training more natural.

In modern deep learning, distributed training is essential for handling large-scale data and models. Typical data parallelism strategies split training batches across multiple GPUs or compute nodes, with each device computing local gradients before global aggregation. However, CLIP's softmax loss requires accessing similarity scores from the entire batch when computing the normalization term, meaning all devices must first exchange intermediate results, perform global normalization calculations, and only then compute loss and gradients. This global dependency causes significant communication overhead and synchronization latency, becoming a scaling bottleneck. In contrast, SigLIP's sigmoid loss only requires information from each sample pair itself, with devices only needing to synchronize gradients during backpropagation, greatly reducing communication requirements. This difference becomes especially apparent when training scales to hundreds of GPUs and billions of samples—SigLIP can scale more linearly, while CLIP's communication costs rise rapidly with device count, limiting effective batch sizes and training efficiency.

Summary: Identical Structure, Training Objectives Determine Performance Differences

The comparison between CLIP and SigLIP profoundly reveals a principle that recurs in modern AI: what determines model capability is often not the architecture itself, but the design of training objectives.

Both share nearly identical dual-tower architectures and embedding mechanisms, with the only difference being the loss function—CLIP uses softmax-based contrastive loss, emphasizing relative competition between samples; SigLIP uses sigmoid-based binary loss, emphasizing independent judgment of each pairing. This change brings two direct benefits: it alleviates the problem of multiple reasonable matches conflicting within batches, and significantly improves distributed training scalability.

For developers building or selecting vision-language models, understanding this difference helps make more informed encoder choices in practical scenarios.

Key Takeaways

• Vision encoders convert images into vector representations processable by VLMs; current mainstream solutions are all Transformer-based
• CLIP and SigLIP are highly similar architecturally; the core difference lies in training objective design
• CLIP uses softmax for contrastive learning, forcing sample competition through global normalization, potentially causing mutual suppression of reasonable matches
• SigLIP treats each image-text pair as independent binary classification using sigmoid, avoiding unnecessary sample competition
• SigLIP's independent judgment mechanism provides better scalability in large-scale distributed training
• Minor changes in training objectives can yield significant improvements in model performance and training efficiency