CAPI-DINO: A Combinatorial Approach to Adding Global Representations to Self-Supervised Learning

CAPI's Strengths and Limitations: Why Global Representations Are Needed

In the field of self-supervised learning (SSL), choosing a method often involves trade-offs between performance, compute, and data scale. Self-supervised learning is a representation learning paradigm that doesn't rely on human annotations — instead, models construct supervisory signals from the data itself to learn meaningful features. Common self-supervised strategies include contrastive learning (e.g., SimCLR, MoCo), masked image modeling (e.g., MAE, BEiT), and self-distillation (e.g., DINO, BYOL). The core value of these methods lies in their ability to learn general, transferable visual representations from massive amounts of unlabeled data, even when labeled data is scarce or expensive to obtain.

A researcher shared their experiment on Reddit: CAPI has become their go-to self-supervised method when compute or labeled data is limited. The reasoning is straightforward — CAPI trains efficiently, converges stably, and fine-tunes reasonably well even on relatively small datasets. CAPI is a self-supervised method focused on patch-level representation learning. Its core idea is to build pretraining objectives at the patch token level of a Vision Transformer, teaching the model to understand and reconstruct relationships between local regions of an image. Unlike masked reconstruction methods like MAE, CAPI emphasizes interactive representation learning between patches rather than simple pixel-level reconstruction. This design gives CAPI notable advantages in training efficiency and convergence stability, since patch-level objectives are typically easier to optimize than global contrastive objectives, and the gradient signals tend to be more uniform.

However, CAPI has one clear shortcoming: it focuses on local / patch-level representations, and therefore cannot directly produce a global image embedding. In a Vision Transformer architecture, the input image is divided into a series of fixed-size patches (e.g., 14×14 pixels), and each patch is linearly projected into a token. Local / patch-level representations refer to the feature vectors corresponding to each of these tokens — they retain spatial position information and are well-suited for dense prediction tasks like semantic segmentation and object detection. A global image embedding, on the other hand, aggregates (pools) all patch tokens into a single vector that represents the semantic content of the entire image. Common pooling approaches include using a [CLS] token or averaging all patch tokens. Global embeddings are indispensable for tasks like image classification, retrieval, and clustering. For many downstream tasks, the lack of a ready-made global representation significantly reduces out-of-the-box usability.

The Inspiration: From DINO v3's Failed Attempt to Reverse Thinking

The motivation for this CAPI-DINO experiment came from an interesting detail. The author heard Tim Darcet mention on a podcast that during the development of DINO v3, the team had tried replacing the iBOT loss with CAPI loss, but ultimately it didn't work out.

iBOT (Image BERT Pre-Training with Online Tokenizer) is a key component used in DINOv2 that combines masked image modeling with self-distillation. Specifically, iBOT randomly masks a portion of patch tokens in the student network's input, then requires the student network to predict the teacher network's output at the masked positions — essentially an online masked token prediction task where the teacher network acts as a dynamic tokenizer. In DINOv2, the iBOT loss is used jointly with DINO's global distillation loss: the former enhances local representation quality while the latter handles global semantic alignment. Tim Darcet's mention of "replacing iBOT loss with CAPI loss" was essentially exploring whether a better local representation learning objective could substitute for iBOT.

This information sparked the author's interest in thinking in reverse: since directly replacing the loss function didn't work, could a different combination strategy succeed — keeping CAPI as-is and stacking a DINO objective on top of the student model's pooled representations?

To understand this approach, it helps to know how DINO works. DINO (Self-DIstillation with NO labels) is a self-supervised learning framework proposed by Meta AI in 2021, based on a self-distillation mechanism. It uses a teacher-student network architecture: the student network receives locally cropped image views, while the teacher network receives globally cropped views. Both share the same network structure but differ in how parameters are updated — the teacher network's parameters are updated via an exponential moving average (EMA) of the student network's parameters. The training objective is to match the student network's output distribution to the teacher network's output distribution, achieved through cross-entropy loss. A notable finding from DINO is that the attention maps of the trained ViT model can automatically segment foreground objects in images, demonstrating strong semantic awareness.

The key insight behind this approach is "addition" rather than "replacement." The model still trains using the standard CAPI patch-level objective while additionally training a DINO head that operates on the pooled global representation. This preserves CAPI's local modeling capability while attempting to fill in its missing global representation ability.

Experimental Setup and Key Results

The author trained this CAPI-DINO combination on the Bio-DINO dataset using a ViT-B/14 model. Bio-DINO is a DINO variant model specifically trained on biodiversity image data, sourced from large-scale species images collected through natural observation platforms (such as iNaturalist). Visual models for the biodiversity domain face unique challenges: visual differences between species can be extremely subtle (e.g., two closely related butterfly species differing only in wing pattern details), while intra-class variation can be very large (e.g., the same species looking dramatically different across growth stages, seasons, and lighting conditions). This makes high-quality local features critical for species identification, which also explains why the author chose this domain to test CAPI. In the author's own words, the results were "okay, not as good as hoped, but not useless either."

Global Representation Linear Probing Performance

On the global representation front, the ViT-B/14 model achieved a linear probing accuracy of 68.7% on the iNat21 dataset. Linear probing is one of the standard protocols for evaluating pretrained model representation quality. The procedure involves freezing all parameters of the pretrained model and training only a linear classifier (i.e., a single fully connected layer plus softmax) on top of the output feature vectors. High linear probing accuracy indicates that the pretrained model has learned high-quality, linearly separable features. iNat21 (iNaturalist 2021) is a large-scale fine-grained species classification dataset containing approximately 2.7 million images across 10,000 species categories. Due to its extremely fine-grained categories and long-tailed distribution, it serves as an important benchmark for self-supervised learning models in the biodiversity domain. A 68.7% linear probing accuracy on this challenging dataset demonstrates that the model did learn meaningful global semantic representations, confirming that the stacked DINO objective achieved the experiment's core purpose.

The Cost to Local Feature Quality

However, there's no free lunch. The author observed that the local feature maps became worse and "dirtier" compared to the original CAPI. This indicates a real trade-off between the local CAPI objective and the added global DINO objective — gaining global representations comes at a cost to local representation quality.

Training Stability Analysis: Challenges of Multi-Objective Joint Training

A prominent phenomenon in the experiment was the change in loss function behavior. CAPI originally had a very smooth training loss curve, but after adding the DINO objective, the training process became noticeably "choppier."

This relates to a classic optimization dilemma in multi-objective joint training. When a model simultaneously optimizes multiple loss functions, different objectives may exhibit gradient conflict — where the gradient direction of one objective is opposite or orthogonal to another, causing the optimizer to oscillate between the two directions and producing fluctuations in the training curve. The loss weight settings directly determine the relative priority of each objective: an objective with too high a weight will dominate the gradient update direction and may suppress learning of other objectives; too low a weight may render that objective essentially ineffective. In recent years, adaptive loss weighting methods (such as Uncertainty Weighting, GradNorm, MGDA) have attempted to automatically balance multi-objective optimization, but their application in self-supervised learning scenarios is still in early exploratory stages.

The author proposed two possible explanations for the training instability:

• The DINO objective interfered with CAPI more than expected
• The DINO objective dominated certain parts of the optimization process

The DINO loss weight used was 0.5, and the author acknowledged this value might be too high. Lowering this weight is likely the top priority for the next round of optimization. This also reminds us that in multi-objective joint training, loss weight tuning is often the make-or-break hyperparameter.

Training Efficiency: CAPI's Core Appeal Is Preserved

Despite adding the extra DINO objective, the most satisfying aspect of this approach is that it still maintains very high training efficiency. The entire CAPI-DINO model consumed only about 16% of the compute used to train Bio-DINO.

In other words, even with the added objective, the approach retains CAPI's most attractive core trait — low compute cost. Additionally, the author observed that fine-tuning performance with very few labeled samples showed considerable promise, with some cases achieving decent results using less data than expected.

Summary: Gains, Losses, and Future Directions for CAPI-DINO

Overall, the gains and losses of this CAPI-DINO self-supervised learning experiment can be summarized as follows:

• Usable global representations: Fills in CAPI's missing global embedding capability
• Still efficient: Compute consumption is only about 16% of Bio-DINO
• Strong low-data fine-tuning performance: Promising potential in few-shot scenarios
• Cost #1: Local feature quality degrades somewhat
• Cost #2: Training stability decreases, with the loss curve no longer smooth

The author believes there is still room to improve the balance between the two objectives, particularly by lowering the DINO loss weight to mitigate interference. This work was completed using the author's own computer vision training library, Birder, and the model weights have been open-sourced on Hugging Face. Birder is a training framework focused on bird and broader biodiversity image recognition that encapsulates domain best practices, lowering the barrier for researchers to reproduce and extend experiments. The model weights are published on the Hugging Face Hub — which has become the de facto standard platform for model sharing, supporting features like Model Cards and automatic inference APIs that allow other researchers to directly load pretrained weights for validation or fine-tuning, greatly facilitating research reproducibility and community collaboration.

The value of this kind of "combinatorial" self-supervised exploration lies not only in the results themselves but also in what it reveals about the interaction mechanisms between different pretraining objectives. The trade-off between local and global representations is a long-standing, practical problem in self-supervised learning, and experiments that transparently share both successes and failures are exactly the nourishment the community needs most.