DINOv2 vs SigLIP: Why the Huge Gap in k-NN Classification? A Guide to Avoiding Vision Encoder Selection Pitfalls

A Puzzling Experimental Result

Recently, a researcher working on an undergraduate thesis posted an intriguing question in Reddit's r/MachineLearning community. Their topic was fine-grained car classification—specifically, distinguishing between different generations of the VW Golf from used-car listing photos.

Fine-grained car classification is an important application of computer vision in the automotive industry. In the used-car market, pricing varies significantly across model generations, and automatic recognition systems can help platforms reduce manual review costs and prevent seller mislabeling or fraud. Academic benchmarks such as Stanford's Cars Dataset (196 classes) and the CompCars dataset have been widely used to evaluate such tasks, while industry has commercial systems like CarNet and AutoVIN. It's worth noting that the Cars Dataset itself contains only 196 categories, and its images are carefully cropped and aligned—there is a significant distributional gap from the real-world scenarios of used-car platforms (cluttered backgrounds, varied angles, complex lighting). This domain shift is precisely the fundamental reason why real-world deployment is often more challenging than academic benchmarks. Domain shift refers to the systematic discrepancy between the statistical distribution of training data and that of deployment-environment data, causing models that perform excellently on academic benchmarks to drop sharply in real-world scenarios. The VW Golf, one of Europe's best-selling models, has iterated through eight generations since its 1974 debut, and the appearance differences between generations are sometimes extremely subtle, making it a highly challenging test subject in the FGVR domain.

The Historical Evolution of Fine-Grained Visual Recognition (FGVR): As an important subfield of computer vision, FGVR's development can be traced back to the bird species recognition challenges of the early 2010s (CUB-200-2011). The earliest methods relied on manually annotated key part locations, using part alignment to eliminate intra-class variation caused by pose and viewpoint changes. Around 2015, the emergence of Bilinear Pooling methods was an important milestone in FGVR; by capturing second-order statistical relationships between local features through the outer product of two feature extraction networks' outputs, it significantly improved performance without requiring part annotations. Entering the Transformer era, visual attention mechanisms naturally possess the ability to focus on discriminative local regions, increasingly blurring the boundary between FGVR methods and general image classification methods, and transfer learning from large-scale pretrained models has gradually become the mainstream paradigm.

The experimental setup was quite simple: use a frozen encoder to extract image embeddings, followed by a weighted k-NN classifier. On a small dataset (175 training images / 132 test images), the three mainstream vision encoders showed a startling gap in performance:

Encoder	k-NN Accuracy
SigLIP2 SO400M	~92%
CLIP ViT-L	~59%
DINOv2 Giant	~41%

Worth noting are the underlying architectural parameters of the three models: SigLIP2 SO400M uses the ViT-SO400M architecture with about 400 million parameters and an output embedding dimension of 1152; CLIP ViT-L has about 300 million parameters and an embedding dimension of 768; DINOv2 Giant has about 1.1 billion parameters and an embedding dimension as high as 1536. DINOv2 Giant has significantly more parameters and a larger embedding dimension than the other two, yet performed worst on the k-NN task—strong evidence that model capacity is not the decisive factor for k-NN retrieval performance.

What puzzled him most was this: DINOv2 Giant, a self-supervised visual foundation model heavily promoted by Meta with a massive parameter scale, came in last on this task—a full 50 percentage points below SigLIP2. Was this really an expected result?

The Root Cause: Embedding Spaces Are Designed for Different Goals

To understand this huge gap, the key lies in grasping the essential differences in the training objectives of the three model types, and how these differences shape their respective embedding spaces.

Contrastive Learning vs. Self-Supervised Learning

The poster had already touched on the core issue. He noted that SigLIP is trained via contrastive learning, so its embedding space is naturally built for cosine similarity.

The Technical Principle of Contrastive Learning: Contrastive learning is a self-supervised or weakly-supervised learning paradigm whose core idea is to construct positive sample pairs (semantically similar pairs) and negative sample pairs (dissimilar pairs), training the model so that positive samples are closer and negative samples are farther apart in the embedding space. Both CLIP and SigLIP use image-text contrastive learning: given a batch of image-text pairs, the model is trained to maximize the cosine similarity of matching image-text embeddings and minimize that of non-matching pairs. From an information-theoretic perspective, this process essentially maximizes the mutual information between image representations and text representations, while using large-scale negative samples to force the model to learn sufficiently fine-grained semantic discrimination.

The Evolution from InfoNCE to Sigmoid Loss: The InfoNCE loss used by CLIP (a softmax-based contrastive loss) requires normalization within a batch: for each image, the model must find the matching one among all texts in that batch—the larger the batch, the more negatives, and the stronger the gradient signal. CLIP's original training used a batch size as high as 32768. This design causes two problems: extremely high computational resource demands, and unstable training signal distribution across batches. SigLIP's improvement replaces softmax with sigmoid activation: each image-text pair is independently judged as matching or not (binary classification), no longer requiring relative ranking within a batch. This change gives each positive and negative pair an independent and equal gradient contribution, thereby achieving a more balanced embedding distribution—namely the isotropy of the embedding space: vectors are uniformly distributed over the high-dimensional hypersphere, and cosine similarity has higher discriminative power. This is precisely the deeper reason for its significant advantage in k-NN retrieval scenarios.

Isotropy is an important metric describing the geometric health of an embedding space. Research shows that many language models and vision models suffer from severe anisotropy problems—a large number of vectors are concentrated in a narrow cone-shaped region of high-dimensional space, causing the cosine similarity between any two vectors to be high and severely reducing discriminative power. SigLIP's sigmoid loss, by independently applying gradients to each sample pair, naturally suppresses this concentration tendency, distributing embedding vectors more uniformly over the hypersphere and providing a more reliable distance metric for k-NN retrieval.

It's worth mentioning that SigLIP2 SO400M is the flagship variant in the vision-language model series released by Google DeepMind in 2024, trained on billions of image-text pairs and introducing NaFlex (Native Resolution Flexible) technology, which supports variable-resolution input, avoiding the information loss caused by traditional fixed-resolution cropping. The design motivation for NaFlex comes from a key insight: distinguishing car generations often relies on details like front headlights and grilles, while traditional fixed square cropping causes vehicle proportion distortion and edge detail loss. Variable-resolution input allows the model to process images at near-original proportions, preserving local details more completely. NaFlex draws on ViT's flexible patch-splitting mechanism in its technical implementation: it dynamically adjusts patch sequence length to accommodate input images of different aspect ratios, while using 2D positional encoding interpolation to preserve spatial structure information—all without retraining, allowing resolution to be adjusted on demand during inference.

DINOv2, by contrast, is a pure-vision self-supervised model with no text supervision signal whatsoever, and no explicit contrastive objective to optimize the geometric property that "similar samples should be close." DINOv2 was proposed by Meta AI in 2023, and its training framework combines multiple self-supervised techniques: with iBOT (Image BERT Pre-training with Online Tokenizer) as its backbone, combined with DINO's self-distillation objective and SwAV's clustering consistency loss, plus a masked image modeling task. In full, DINOv2's training includes three levels of self-supervised objectives: (1) the global image-level DINO self-distillation loss, which forces the model to learn view-invariant global representations through a teacher-student network; (2) the local patch-level iBOT masked image modeling loss, similar to BERT in the visual domain, requiring the model to reconstruct masked image patches; (3) SwAV-style prototype clustering consistency constraints, enhancing the clustering separability of features. This multi-task joint training gives DINOv2 extremely strong local feature perception—its patch-level features can be directly used for dense prediction tasks like semantic segmentation, depth estimation, and object detection, even without any fine-tuning. However, precisely because the training objectives focus on pixel-level and region-level visual understanding rather than explicitly optimizing the separability of global vectors, its CLS token embedding space does not form the clear semantic clustering structure of contrastive learning models, inherently limiting "out-of-the-box" k-NN retrieval.

Here it's necessary to specifically explain the role of the CLS token in DINOv2. In the Vision Transformer architecture, the CLS token is a special learnable vector fed in alongside the image patch sequence; after interacting with all patch tokens through multiple layers of self-attention, it is used as the global representation of the entire image. The DINO series of models specifically optimizes the consistency of the CLS token through a teacher-student distillation framework, but this optimization objective is that "the CLS tokens under different views should be consistent with each other" rather than "the CLS tokens of different semantic categories should be separable from each other"—the two are not mathematically equivalent, and this is precisely the deep architectural reason for DINOv2's limited performance in k-NN retrieval.

An analogy that helps intuitive understanding: DINOv2's embedding space is more like an extremely rich encyclopedia—enormous amounts of information, but you need to know how to search to find the answer you want; whereas SigLIP's embedding space is like a carefully labeled library index system—the distance relationships between entries themselves directly reflect semantic similarity, and you can find similar content just by browsing.

Why L2 Normalization Didn't Save the Day

The poster once suspected the issue was the choice between cosine distance and Euclidean distance. But he quickly ruled out this guess: the embeddings had already been L2-normalized, and both distance metrics gave completely identical ranking results. In other words, the problem is not the distance metric, but the separability of the embedding space itself.

This is the core insight: SigLIP has already encoded "semantic separability" into the angular relationships of the vectors, so bare k-NN can reach 92% accuracy; DINOv2's raw features, however, require a trained classification head (such as a linear probe) to "translate" and recombine them before their true discriminative power can be unleashed.

The Essential Difference Between Linear Probe and k-NN: These two evaluation methods make fundamentally different assumptions about the embedding space. k-NN is a completely parameter-free method that directly relies on the distance relationships between embedding vectors for classification, requiring the embedding space itself to have good semantic clustering structure—that is, samples of the same category naturally cluster together in high-dimensional space. Notably, in small-sample scenarios (such as the 175 training images in this case), k-NN also faces the challenge of the Curse of Dimensionality—the discriminative power of distance metrics in high-dimensional embedding spaces decays as dimensionality increases, and the sensitivity to k-value selection is also amplified. The essence of the curse of dimensionality is: in high-dimensional space, distances between data points tend to become uniform, and the difference (i.e., contrast) between the distances to the nearest and farthest neighbors decays exponentially, causing the reliability of distance-based classification methods to plummet. Take DINOv2 Giant as an example: its output embedding dimension is as high as 1536, while there are only 175 training samples, meaning each dimension has on average fewer than 0.1 training samples providing statistical support—greatly compromising k-NN's reliability. Linear Probe, on the other hand, adds a trainable linear classification layer on top of frozen features, finding the optimal decision hyperplane through supervised training, able to re-"calibrate" the geometric structure of the feature space and thereby mine hidden discriminative information within the features. This is also why DINOv2 performs far better in linear probe evaluations than in k-NN: its features contain rich visual information, but require a trained linear layer to "translate" this information into classification decisions.

Is DINOv2 Really Unsuitable for Retrieval Tasks?

This is the most worthwhile point to expand on in the entire discussion. The answer is: it depends on how you use it.

Frozen Features + k-NN Is an "Unfair" Evaluation of DINOv2

Running k-NN directly with frozen DINOv2 global features essentially tests its unadjusted embedding space geometry—which happens to be DINOv2's relatively weakest evaluation method. DINOv2's strong official-paper results are mostly based on linear probes or more complex downstream task heads. With a trainable linear layer added, DINOv2 can often clearly catch up to or even surpass contrastive learning models on many fine-grained tasks.

The Special Nature of Fine-Grained Recognition Tasks

Fine-Grained Visual Recognition (FGVR) is a classic challenge in computer vision, aiming to distinguish subcategories within the same base category. Distinguishing between different Golf generations is a typical example of this kind of task—model differences may only manifest in local details such as front-end lines, headlight shapes, and grilles. The core challenge of these tasks is: inter-class variance is low, while intra-class variance is high—the difference between photos of the same Golf generation under different lighting, angles, and shooting distances may far exceed the appearance differences between two adjacent generations.

FGVR methods in the deep learning era have undergone clear technical evolution: early methods relied on explicit part detection (Part-based Models), aiding classification by annotating the locations of key parts such as headlights, wheels, and grilles; subsequently, Bilinear Pooling methods captured high-order interactions of local features through the outer product of two feature extractors, achieving significant improvements without part annotations; the introduction of attention mechanisms enabled models to automatically focus on discriminative regions; while the current large-model era relies more on powerful pretrained features combined with lightweight adaptation layers. Classic FGVR methods often need to explicitly capture local discriminative regions (such as part-level attention mechanisms) rather than rely on a single global vector.

In theory, DINOv2's powerful local feature representation should give it an advantage in such tasks. But if you only take the CLS token for global pooling, this local discriminative information may be diluted, leaving the model unable to shine.

Practical Advice for Vision Encoder Selection

Synthesizing the above analysis, developers using vision encoders for fine-grained classification or image retrieval should keep several points in mind:

1. First, Clarify Your Usage Paradigm

If you pursue "zero-training, out-of-the-box" retrieval performance, contrastive learning models (SigLIP, CLIP) are the safer choice, as their embedding spaces are naturally suited to similarity retrieval. SigLIP2 SO400M's 92% performance in this case fully confirms this.

2. Don't Skip This Step When Using DINOv2

If you insist on using DINOv2, be sure to pair it with a linear probe or lightweight classification head before evaluating. You can also try:

• Using intermediate-layer features rather than the final layer
• Trying different pooling strategies (average pooling over patch tokens, rather than only taking the CLS token)
• Fusing local patch features for fine-grained tasks is often more effective than a single global vector

A worthwhile direction to try in practice: feed DINOv2's patch token feature map (rather than the CLS token) into a lightweight attention aggregation module (such as Cross-Attention Pooling), letting the model automatically learn how to distill a discriminative global representation from local features. This "feature aggregation head" has an extremely small number of parameters (usually no more than a few hundred thousand), so it's unlikely to overfit even in the extreme small-data scenario of 175 training samples, while fully unleashing the potential of DINOv2's local features. Cross-Attention Pooling works as follows: it introduces a set of learnable query vectors that interact with the patch token sequence through an attention mechanism, dynamically assigning weights to features at different spatial locations—for the car-generation recognition task, the model will gradually learn to focus attention on generation-discriminative regions such as the front grille and taillight shape, while ignoring interfering information such as body color and background environment.

3. Additional Considerations for Small-Dataset Scenarios

175 training samples is a fairly limited scale, actually corresponding to the Few-Shot Learning scenario in machine learning research. The core challenge of Few-Shot Learning is how to generalize from very few labeled samples to unseen samples. Classic methods include Prototypical Networks, Matching Networks, and Relation Networks. The Prototypical Networks approach is especially relevant to this case: it computes the mean vector of the support-set sample embeddings for each category as a "prototype," then classifies a query sample into the category of the nearest prototype by Euclidean distance—this is essentially a variant of k-NN, but by computing category prototypes rather than directly comparing individual samples, it effectively reduces the impact of noisy samples in high-dimensional space. Research shows that in such scenarios, the clarity of the semantic clustering structure of pretrained features is more important than the number of downstream trainable parameters.

When data is extremely scarce, the core consideration in model selection shifts from "feature representation capability" to "transfer learning efficiency"—that is, the degree of alignment between pretrained features and the target task distribution. SigLIP is trained on massive image-text pairs, so its embedding space is already well semantically structured; even for unfamiliar fine-grained subcategories, it can distinguish them by relying on relatively consistent visual-semantic relationships. DINOv2's potential requires sufficient downstream data to activate the parameters of its trainable layers, and this condition is hard to meet in small-sample scenarios.

From the domain adaptation perspective of Transfer Learning Theory, this phenomenon can be clearly explained: the more aligned the distributions of the source domain (pretraining task) and target domain (fine-grained k-NN retrieval), the fewer target-domain samples are needed. SigLIP's pretraining objective is essentially the same problem as the k-NN retrieval task (similarity ranking), so transfer efficiency is extremely high; DINOv2, however, has a clear task mismatch and needs more downstream data to bridge this gap. This also corresponds to the importance of inductive bias: the inductive bias of contrastive learning models manifests as "semantically similar images are closer in embedding space," and this prior is explicitly encoded into the model weights through massive image-text pairs; DINOv2's inductive bias tends toward "image regions with consistent visual content share local features," which is better suited to dense prediction tasks.

Inductive Bias is a core concept in machine learning, referring to the set of prior assumptions a model uses to generalize when there is insufficient training data. In the deep learning context, inductive bias comes both from architectural design (such as CNN's translation invariance and Transformer's global attention) and from training objectives (such as contrastive learning's "similar samples cluster together" prior). An inductive bias highly matched to the target task can often provide critical generalization ability when data is insufficient, and its value in small-sample scenarios can even exceed model capacity itself. In this case, during pretraining SigLIP already wrote the prior "similar images are semantically close" into the model weights, so 175 samples were already sufficient for k-NN to work; DINOv2's inductive bias is closer to "visual reconstruction" than "semantic retrieval," and with the mismatch between task and prior, 175 samples are insufficient to transform its potentially rich features into accurate classification decisions via k-NN. This "leverage effect" of inductive bias is especially pronounced when data is scarce: a prior assumption highly matched to the task can boost effective sample utilization by several to dozens of times, far exceeding the benefit of increasing the number of model parameters. When data is insufficient, an inductive bias better matched to the task is often more valuable than greater model capacity.

There Is No Best Model, Only the Most Suitable Usage

This Reddit post may look like a "crash," but it actually reveals a common misconception in vision foundation model selection: large parameter count and big reputation ≠ strongest in any paradigm.

DINOv2 and SigLIP represent two different technical routes, serving different downstream needs. SigLIP was born for cross-modal alignment and retrieval; DINOv2 was born for dense visual understanding. When evaluated with k-NN "bare retrieval," the former has all the advantages; but this does not mean the latter is "worse"—it's just that the evaluation paradigm and model design goals were mismatched. Such mismatches are extremely common in industrial practice: engineers often make selection decisions based on a model's ranking on comprehensive leaderboards, while ignoring the differences in assumptions behind different evaluation paradigms. Understanding this point has more long-term value than memorizing any specific accuracy number.

For this graduate student, perhaps the most valuable takeaway is not "which model has higher accuracy," but truly understanding how an encoder's training objectives determine the geometric properties of its embedding space, and how these properties affect the choice of downstream task paradigm. This is the core intuition that needs to be internalized over the long term in deep learning engineering practice.

Key Takeaways

Key Takeaways