A Guide to Avoiding Pitfalls in PCA Visualization of Self-Supervised Vision Models: Why the Colors Are Meaningless

Running Self-Supervised Vision Models on a Laptop CPU

Many people assume that understanding the features of Self-Supervised Learning (SSL) vision models requires expensive GPU clusters. But one developer proved that it can be done with CPU alone, using a MacBook (M-series chip) equipped with 64GB of RAM.

What is self-supervised vision learning? SSL is a paradigm for training models without manually annotated labels. The rise of this paradigm stems from a rethinking of the cost of large-scale human annotation—labeling ImageNet consumed millions of hours of human labor, and the goal of self-supervised methods is precisely to let models extract supervisory signals from the data itself. In the vision domain, representative methods include DINO (2021, Meta AI, based on knowledge distillation and self-attention consistency), MAE (Masked Autoencoders, 2022, Kaiming He's team, which randomly masks 75% of image patches and reconstructs the pixels), and MoCo (Momentum Contrast, 2020, which maintains a dynamic dictionary for contrastive learning). By designing "pretext tasks," these methods let models extract supervisory signals from images themselves—for example, contrasting different cropped views of the same image, forcing the network to learn semantically consistent features. What these three types of methods have in common is that they can learn semantically rich feature representations without labels, and their transfer performance on downstream tasks already approaches or even exceeds that of supervised pretraining. After training, such models often spontaneously acquire the underlying structures needed for tasks like semantic segmentation and depth estimation, even though they never saw labels for these tasks during training. This is exactly why visualizing SSL features is so appealing: people want to "see" what the model has actually learned.

He used the ViT-S checkpoint provided by the LingBot-Vision project (Apache-2.0 license). This model has only about 21 million parameters, with a weight file of just 86.5MB, and processing a single 512×512 image at fp32 precision on CPU takes about 1.4 seconds. Setting up the entire environment took only 35 seconds to install dependencies and 8.4 seconds to download the weights. Such a low barrier to entry allows anyone to explore what SSL features actually look like on their own computer.

On the acceleration potential of M-series chips: Apple M-series chips (M1/M2/M3/M4) use a System-on-Chip (SoC) design that integrates the CPU, GPU, Neural Engine (ANE), and memory onto the same chip, forming a Unified Memory Architecture (UMA). In traditional x86 + discrete GPU architectures, data must be copied back and forth between CPU memory and video memory via the PCIe bus (bandwidth 16-32 GB/s); whereas the CPU and GPU on M-series chips share the same memory pool, with bandwidth reaching 100-400 GB/s (increasing with the model), eliminating this transfer bottleneck. PyTorch's MPS (Metal Performance Shaders) backend has supported M-series GPU acceleration since version 1.12. Choosing --device cpu in the article is the most conservative compatibility option; in practice, using --device mps can usually accelerate things by another 3-5x, but note that the MPS implementations of some operators (such as certain attention variants and specific normalization layers) still have precision discrepancies, so it's recommended to use CPU results as a baseline for comparison before critical tasks.

How PCA Visualization Works

To understand this experiment, we first need to understand how the visualization works. A ViT (Vision Transformer) outputs a 384-dimensional feature vector for each 16×16 image patch.

ViT's Patch Mechanism: The Vision Transformer was proposed by Google Brain in the 2020 paper An Image is Worth 16x16 Words, completely breaking the monopoly of convolutional networks over vision tasks. Its core innovation lies in splitting the input image into fixed-size patches, each of which is flattened and linearly projected into a one-dimensional token, serving as the input to the Transformer. Unlike convolutional networks that rely on local receptive fields, the Transformer allows each token to interact with all other tokens through the self-attention mechanism, modeling global context—this is the fundamental reason DINO features can spontaneously develop segmentation ability under unsupervised conditions: the global context naturally produces a clustering structure in the token distribution between foreground and background. The ViT-S (Small) version has an embedding dimension of 384 and a total of 12 Transformer layers, meaning each token corresponds to a 384-dimensional feature vector representing the semantic encoding of that image region after global information interaction.

For a 512×512 image, this forms a 32×32 grid, totaling 1024 tokens.

The author applies PCA dimensionality reduction to these 1024 vectors, retaining the top 3 principal components, and then maps each patch to an RGB color.

The mathematical principle of PCA dimensionality reduction: The goal of Principal Component Analysis (PCA) is to find the projection directions with the maximum variance in the data. In this scenario, PCA performs Singular Value Decomposition (SVD) on the 1024×384 token matrix, and the top 3 right singular vectors define a three-dimensional projection subspace. Each patch's 384-dimensional vector is projected onto these 3 principal components, yielding 3 scalars that are respectively mapped to the color values of the RGB three channels. The key limitation is: due to the sign ambiguity in eigenvalue decomposition, the coordinate system direction can be arbitrarily rotated or even flipped between different images or different resolutions. The principal component directions of PCA are fit on the token distribution of the current image, and the "first principal component" of different images points in different directions, so the colors are completely incomparable across images.

The resulting visualization can clearly separate the two chickens, the wire mesh, the grass, and the dirt in the scene into their own coherent color blocks with sharp boundaries—it looks just like a semantic segmentation mask.

And it is precisely this that becomes the beginning of misinterpretation.

Colors Are Not Class Labels

Beginners who see such an image naturally think "red represents chickens, blue represents the fence." But the author overturned this understanding with a simple control experiment: using exactly the same weights and exactly the same photo, he only changed the input size from 512 to 1024.

As a result, the colors were completely reversed. The chickens went from magenta and red to green and cyan, and the fence background went from pure blue to yellow spots.

The reason is: PCA is fit separately for each image, based on that image's own token distribution. The resolution change caused the number of tokens to increase from 1024 to 4096, the entire distribution changed, and the directions of the principal components rotated accordingly. To obtain a consistent color mapping across images, one would need to fit a global PCA on a large image set, or switch to dimensionality reduction methods like UMAP that preserve local topological structure, projecting the tokens of all images into a unified coordinate system. Therefore, you cannot fixedly interpret "red" as "chicken"—colors do not carry cross-image semantic information, they only reflect the relative difference structure within that image.

Only "grouping" is meaningful, never "hue."

Higher Resolution Does Not Equal Better Results

Another counterintuitive finding is that increasing resolution changes what the tokens encode, rather than just making the results more refined.

At 1024 resolution, the originally clear wire mesh background instead dissolved into high-frequency yellow spots. The mechanism behind this is: ViT uses fixed 16×16 pixel patches, and when the input is increased from 512 to 1024, the same physical scene is split into more and smaller patches—each metal wire of the mesh went from spanning several patches to being distinguishable within a single patch, and the model's receptive field descended from "overall background texture" to "the material of a single metal wire," meaning the semantic granularity underwent a fundamental shift, rather than a simple improvement in image quality.

This reminds us that input size is a hyperparameter that substantially affects feature semantics, not just an image-quality knob. When choosing resolution in practical applications, one needs to first clarify the "target semantic granularity": use 512 to segment overall object contours, use 1024 to analyze fine textures, rather than blindly pursuing high resolution.

Transparent Objects Remain a Blind Spot

The author also tested a photo of a plastic box containing transparent glassware. As a result, the transparent glass basically "melted" into the background color region.

The technical root of the challenge with transparent objects: The perception of transparent and translucent objects is a long-standing, not fully solved problem in computer vision. Transparent materials transmit light rather than reflecting it, causing the pixel colors on the object's surface to be determined by the background rather than by the object's own material properties—self-supervised models trained on boundary and texture cues rely on the "statistical regularities of object surfaces," but transparent objects have almost no stable texture features on their surfaces. Academia has established dedicated benchmarks for this problem: Trans10K (2020) contains 10,000 images of transparent objects, and ClearGrasp (2019, Google) focuses on depth estimation of transparent objects in robotic grasping. Mainstream engineering solutions include polarization cameras (using the polarization properties of refracted light to distinguish transparent surfaces), structured light or ToF sensors (directly measuring time-of-flight to obtain depth, without relying on color information), and the depth-completion approach mentioned in the article—bypassing this perception difficulty by fusing RGB images with the sparse point cloud from a ToF camera, which is the most mature compromise strategy in current engineering practice.

Larger Models Are Not as Important as Imagined

The author also tested the larger ViT-L (about 300 million parameters, with a checkpoint of 1213MB, taking 4.5 seconds per image on CPU).

Comparison of parameter scales: The ViT series is divided into several variants by scale: ViT-S (embedding dimension 384, 12 layers, ~22 million parameters), ViT-B (embedding dimension 768, 12 layers, ~86 million parameters), and ViT-L (embedding dimension 1024, 24 layers, ~300 million parameters). Larger models have higher feature capacity and tend to learn smoother feature manifolds, so the output images have fewer noise spots; but this does not mean a linear improvement in semantic understanding ability.

The results show that the output of the larger model is smoother with fewer spot artifacts, but the overall structure is basically consistent.

ViT-S is already able to correctly separate objects. If your goal is only to understand the structure of SSL features, there is no need to resort to large models at all—this is good news for teaching, prototype validation, and low-cost exploration. This phenomenon is consistent with the conclusion of "over-parameterization" in neural network research: once the model capacity exceeds the threshold needed to capture the semantic structure of a scene, the main contribution of the extra parameters is the smoothness of the feature manifold (reducing noise), rather than adding new semantic distinguishing ability. This finding overturns the intuition in classical statistical learning theory that "higher complexity means greater risk of overfitting" (corresponding to the double descent curve theory), and in scenarios with limited computational resources, small models are often a more reasonable starting point.

Practical Pitfalls: Two Details That Easily Trip Up Newcomers

The author also shared two common traps:

The default device is CUDA: The demo script defaults to --device cuda, which on a Mac will immediately crash with a "CUDA not compiled" assertion. You must explicitly pass --device cpu or --device mps, but the quick-start section of the README doesn't mention this at all.
--ckpt requires the full path: This parameter requires the full path returned by hf_hub_download, not a bare model.pt placed in the working directory, otherwise it will raise a FileNotFoundError.

Complete reproduction command (run from the repository root after pip install -r requirements.txt):

CKPT=$(python -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('robbyant/lingbot-vision-vit-small','model.pt'))")
python -m lingbot_vision.pca_demo \
  --ckpt "$CKPT" \
  --config-file lingbot_vision/configs/lbot_vision_vits.yaml \
  --input examples/example.png \
  --out out \
  --device cpu \
  --dtype fp32 \
  --size 512

The code repository is located at github.com/robbyant/lingbot-vision, and the weights are hosted under the robbyant/lingbot-vision collection on Hugging Face.

Conclusion: Understand Visualizations So They Don't Deceive You

The most valuable takeaway from this experiment is not the engineering conclusion that "CPU can also run SSL models," but the clarification of the essence of visualization. PCA-colored images look like segmentation results, but their colors are the product of independent fitting for each image and lack cross-image semantic consistency. To achieve cross-image comparable visualization, one needs to fit a global coordinate system on a large-scale dataset, which is fundamentally different from the lightweight exploration path shown in this article.

For anyone researching and using vision features, this is an important reminder: Visualization is a window for understanding models, but the visual intuition it carries may mislead you. Colors are merely the relative-difference projection of the token distribution within that image, not absolute class labels. Focusing only on structure and grouping, ignoring specific hues, is the correct way to read these images.

Key Takeaways

Self-supervised vision models (DINO/MAE/MoCo) can learn semantically rich features without labels, and the small ViT-S (~21 million parameters) can run on a laptop CPU
The colors in PCA visualizations carry no cross-image semantic information; each image is fit independently, and colors only reflect the relative differences in the token distribution within that image
Resolution is a key hyperparameter affecting semantic granularity, not just an image-quality knob; higher resolution makes the model switch from perceiving the whole to perceiving fine texture details
ViT-L and ViT-S differ little in structural perception, confirming the "over-parameterization" theory: extra parameters mainly improve feature smoothness rather than adding semantic ability
Transparent objects are an inherent blind spot of purely visual self-supervised methods, requiring polarization cameras, ToF sensors, or multimodal fusion for reliable handling