Small-Scale Replication of "Good Views Beat Next-Token Prediction": Holds Up with Human Descriptions

LeCun et al.'s "good views" training objective claims to align text models with image models faster. But the largest experiment in the original paper used text generated by an automatic image captioner, meaning image information could leak through the descriptions. A developer shared his small-scale replication on Reddit: using only human-written COCO captions and training an 11-million-parameter text encoder on a laptop, the results still support the original conclusion.

Experimental Design: Control the Variables, Look Only at Training Pairs

The core idea of the replication is to keep everything unchanged and only alter how the text-side "views" are constructed. The data comes from 10,000 COCO photos, each with 5 human captions, split 8000/1000/1000 into train, validation, and test sets. The image side uses a frozen DINOv2-base, embedded only once; the text side uniformly uses a 4-layer, width-256 encoder with about 11 million parameters.

The three training objectives share the same caption text, 3000 training steps, batch size 256, and the same anti-collapse loss (VICReg-style variance and covariance terms, rather than the original paper's SIGReg). The text model never sees images during training. The evaluation metric is the mutual 10-nearest-neighbor matching rate between caption embeddings and DINOv2 embeddings on 1000 test photos, with random chance at 0.010.

This design isolates "view quality" as the only variable, making it cleaner than the original paper—an automatic captioner might write object and scene information from the image directly into the text, letting the text model "cheat."

Mutual nearest-neighbor matching rate is a metric for evaluating the alignment quality of cross-modal embeddings. Given the DINOv2 embedding of each image and the text embedding of each caption in the test set, for each text embedding, take its nearest neighbor in the similarity ranking against all image embeddings; then for each image embedding, take its nearest-neighbor text. Only when the two are mutual nearest neighbors is it counted as a successful match. The matching rate is the number of successful pairs divided by the total number of samples. Under random chance, the probability of a random match among 1000 samples is about 0.010, consistent with the text. This metric is sensitive to the global structure of the embedding space, but is looser than retrieval R@1 because it does not require ranking first—only mutual nearest neighbors.

DINOv2-base is a self-supervised vision Transformer model proposed by Meta, based on the ViT-B/14 architecture with about 86 million parameters. It is pretrained on large-scale unlabeled images via a self-distillation objective, producing high-quality general-purpose visual features without text supervision, and performs well on downstream tasks such as classification, segmentation, and retrieval. In this replication, the image side uses a frozen DINOv2-base, meaning image embeddings remain unchanged throughout training, and the text encoder learns to map text to positions aligned with this fixed visual space. This design eliminates variables introduced by fine-tuning the image encoder, focusing the experiment on the effect of the text-side training objective.

Results: Good Views Lead, Ordering Stable

Averaged over 3 random seeds at the same learning rate, good views (two different captions of the same photo) reach 0.320, bad views (a single caption with some words masked) 0.192, and next-token prediction 0.148. The gaps are clear, with good views' matching rate more than double that of next-token prediction.

The author also did a small learning-rate search for each method, taking the best rate and a single-seed run, yielding 0.328 / 0.196 / 0.168, with the ordering unchanged. This shows the lead is not an accidental product of learning-rate tuning.

Retrieval metrics (R@1, with a fitted rotation) show a narrower gap: 0.084 vs. 0.063 vs. 0.057. The narrowed gap may mean the mutual nearest-neighbor metric amplifies the advantage of good views, but the direction remains consistent.

R@1 (Recall at 1) is a standard metric in retrieval tasks: for each query, return the top-ranked result; if it matches the ground-truth match, count it as a hit; the number of hits divided by the total number of queries is R@1. The mention of "with a fitted rotation" means that before computing the retrieval metric, a linear transformation (rotation matrix) is applied to the text embeddings to correct the global geometric difference between text space and image space—a common post-processing step in cross-modal retrieval evaluation. R@1 is stricter than the mutual nearest-neighbor matching rate because it only considers the top-ranked candidate, so its values are usually lower. The narrower gap between good views and next-token prediction under R@1 suggests the mutual nearest-neighbor metric may be more sensitive to the advantage of good views.

Limitations and Open Questions

The replication scale is far smaller than the original paper: 11 million parameters vs. 109 million or even 72 billion, and 8000 training photos is far fewer than the original setup. The good-views method requires knowing which captions correspond to the same photo, which relies on annotators writing while looking at the photo—in real applications, such pairing information may not always be available.

The experiment covers only one dataset and one image model, and the best learning rate for next-token prediction happens to fall at the upper bound of the search range, so better unexplored configurations may exist. The author also notes that the additional learning-rate search step was added after seeing the first-round results, and this is noted in the code repository.

The most critical open question is: as encoder scale increases, will the gap between good views and next-token prediction shrink or grow? A laptop cannot answer this. The author has open-sourced the code, figures, and results of each run in the GitHub repository (IFAKA/platonic-lab), and publicly asks whether anyone has done a larger-scale validation.

VICReg-style anti-collapse loss is a regularization term used in the VICReg (Variance-Invariance-Covariance Regularization) method to prevent embedding space collapse, comprising variance and covariance terms. The variance term requires the variance of each embedding dimension to stay above a threshold, preventing all samples from being mapped to the same point; the covariance term requires covariances between different dimensions to be near zero, encouraging each dimension to carry non-redundant information. The original paper uses SIGReg (Sketched Isotropic Gaussian Regularization), whose goal is to pull the embedding distribution toward an isotropic Gaussian. The replication switches to a VICReg-style loss to simplify implementation while preserving the anti-collapse function, and to avoid introducing the original paper's specific regularization details, making variable control cleaner.

Significance for the Representation Learning Discussion

The value of this small-scale test is not in overturning or confirming the original paper's large-scale conclusions, but in ruling out a specific alternative explanation: automatic captioners leaking image information. When the text is entirely human-written and the model has only 11 million parameters, good views still lead, showing that the idea of "constructing training pairs from multiple views of the same instance" has independent value and does not fully depend on caption quality or model scale.

The takeaway for practitioners is that constructing multi-view paired data may be cheaper than imagined—human descriptions are naturally diverse, and different captions of the same photo are ready-made "good views." But the prerequisite is preserving the pairing relationship of "which captions belong to the same instance," which is easy at small scale but an unresolved engineering problem on large-scale web data.

One-line Summary

A small-scale replication confirms that the good-views training objective still outperforms next-token prediction with purely human descriptions, ruling out the image-information-leakage explanation.

Paragraph Summary

A developer shared on Reddit a small-scale replication of LeCun et al.'s "good views" training objective. The original paper's largest experiment used text generated by an automatic image captioner, so image information could leak through the descriptions. The replication uses only human-written COCO captions, trains an 11-million-parameter text encoder on a laptop, and uses a frozen DINOv2-base on the image side. The three training objectives (good views, bad views, next-token prediction) share the same text, steps, batch, and anti-collapse loss, changing only how training pairs are constructed. Results show good views achieve a mutual nearest-neighbor matching rate of 0.320, significantly outperforming bad views at 0.192 and next-token prediction at 0.148, with the ordering unchanged after learning-rate search. Under the retrieval metric R@1, the gap narrows, suggesting the mutual nearest-neighbor metric may amplify the advantage. The scale is far smaller than the original paper, and how the gap changes with larger encoders remains an open question. Code and results are open-sourced.