[KongchangAI]
· 2 min read· 1,320 words

Meta & NYU: 200+ Multimodal LLMs Prove Vision Encoders Are Not Required

Meta & NYU: 200+ Multimodal LLMs Prove Vision Encoders Are Not Required

200+ from-scratch MLLMs show vision encoder pretraining is optional, but vision-specific parameters are essential.

Meta FAIR and NYU trained over 200 MLLMs from scratch under unified data, compute, and design variables. Under matched compute, pretrained encoders at best tie with from-scratch joint training; encoder-free models need ~3x compute to catch up. Their in-fusion approach embeds vision weights in early LLM layers, matching separate encoders while cutting time to first token by ~13%.

Title

Meta & NYU: 200+ Multimodal LLMs Prove Vision Encoders Are Not Required

Meta Description

Meta FAIR and NYU trained over 200 multimodal LLMs from scratch and found under matched compute that a pretrained vision encoder is not required, but vision-specific parameters are essential; the in-fusion approach saves about 13% in time to first token.

Article Content

Are pretrained vision encoders mandatory for multimodal large language models (MLLMs)? A large-scale controlled study from Meta FAIR and NYU delivers a counterintuitive answer: a pretrained vision encoder is not necessary, but vision-specific parameters themselves are indispensable. The study trained over 200 MLLMs from scratch, systematically disentangling the role of visual encoding in MLLMs under tightly controlled data, compute, and design variables.

Reddit discussion thread screenshot

Research Background and Motivation

Current model vendors have diverged noticeably in their choices around vision encoders. As the authors note in their Reddit post, Gemma 4 12B dropped the encoder entirely, Kimi K3 abandoned encoder pretraining, while Inkling 2 Small brought the encoder back. Published models differ in training data, compute scale, and whether encoder pretraining compute is counted in the total budget, so these public results cannot truly settle the debate.

To address this, the authors ran a large controlled experiment: over 200 models trained entirely from scratch (including the language model portion), using exactly the same data, counting encoder pretraining compute within the same lifecycle total compute budget, and changing only one design choice at a time. The evaluation metric is VQA performance after instruction tuning, with model sizes ranging from 1B to 6.4B.

Vision Encoder is the module in an MLLM responsible for converting images into feature sequences, typically a ViT such as CLIP's or SigLIP's. Traditional MLLM architectures (e.g., LLaVA) connect a pretrained vision encoder to an LLM via a projection layer; the encoder is usually pretrained separately on image-text pairs via contrastive learning and then frozen or fine-tuned when attached to the LLM. The phrase "whether encoder pretraining compute is counted in the total budget" refers to this: if encoder pretraining consumes substantial compute but that compute is not included in a fair comparison against a from-scratch model, conclusions will be biased toward pretrained encoders. This paper's controlled experiment folds that compute into the same lifecycle budget, placing both routes on the same starting line.

Key Finding 1: Encoder Pretraining Is Optional

The study tested four types of pretrained encoders, using pretraining objectives such as contrastive learning, self-supervised learning, and image captioning. Under matched compute, these pretrained encoders at best only tied with encoders trained from scratch and jointly with the LLM.

More interestingly, even with pretraining compute entirely excluded from the budget, extending MetaCLIP's pretraining only raised ImageNet linear probing accuracy from 80.4% to 84.4%, with no improvement in VQA. This suggests that the benefits of vision encoder pretraining show up mainly in pure visual recognition tasks and do not automatically transfer to multimodal understanding.

ImageNet linear probing is a classic method for evaluating visual representation quality: freeze the encoder to extract image features, train only a linear classifier on top, and measure accuracy over ImageNet's 1000 classes. It measures the linear separability of features by object category—a pure visual recognition capability. VQA (Visual Question Answering) instead requires the model to combine image content with a natural-language question to produce an answer, testing cross-modal alignment and reasoning. The two evaluate different dimensions: higher linear probing accuracy does not necessarily yield higher VQA, which is the evaluation-theoretic basis for the paper's claim that "pretraining gains stay in pure visual recognition and do not automatically transfer to multimodal understanding."

Key Finding 2: Vision-Specific Parameters Cannot Be Skipped

Encoder-free models lag behind encoder-equipped versions at every scale, roughly requiring about 3x the compute to catch up. This clarifies a common misconception: the total parameter count itself is not what matters; the role of vision-specific parameters cannot be replaced by "giving those parameters to the LLM."

Experiments show that transferring the encoder's parameter budget to the LLM does improve text benchmark scores, but VQA performance does not improve as a result. In other words, visual information needs dedicated capacity to carry it.

in-fusion: A Middle Ground

Since the parameters must be "vision-specific" but do not necessarily need to form a separate encoder module, the authors propose the in-fusion approach: adding vision-specific weights into the LLM's early layers. At scales from 1.7B to 6.4B, in-fusion performs comparably to a separate encoder while reducing time to first token by about 13%.

Time to First Token (TTFT) is a key latency metric in inference serving, referring to the time from when a user sends a request until the model outputs the first token, mainly determined by the compute in the prefill stage. In a separate-encoder architecture, the image must first pass through the encoder's forward computation and then be fed into the LLM via the projection layer, forming a serial two-stage computation; in-fusion embeds vision-specific weights directly into the LLM's early layers, so visual and text tokens can be processed in the same forward pass, reducing the overhead of a separate encoder forward pass and cross-module transfer—hence lower TTFT. For interactive multimodal applications, TTFT directly affects user experience, which is in-fusion's practical value beyond matching quality.

Practical Implications

For teams with limited resources, the significance of this study is: do not obsess over obtaining or reproducing a pretrained vision encoder—concentrate compute on joint training with the LLM. But do not simply remove vision-specific parameters to simplify the architecture either—that would require about 3x the compute to compensate. in-fusion offers an approach that balances quality and inference latency.

The authors also verified the robustness of their conclusions: when adding vision capabilities to a pretrained 1.7B LLM, the above conclusions still hold.

Limitations and Discussion

The study's controlled nature is its strength, but it also means the conclusions rest on a specific data distribution and training recipe. The authors disclosed their authorship in the Reddit post and opened the paper, blog, and FAQ for community discussion. For researchers and engineers focused on MLLM architecture design, the value of this work lies in breaking the vague debate over "whether an encoder is necessary" into two measurable questions: is pretraining necessary (no), and is vision-specific capacity necessary (yes).

Paper: https://www.alphaxiv.org/abs/2611.17334 Project blog and FAQ: https://unbundling.github.io

Compute-matched comparison is the core of this paper's methodology: comparing different architectures under a fixed total training FLOPs budget, rather than fixing parameter count or training steps. This setup avoids the confound of "giving one approach more compute so it looks better," but it also means conclusions depend on the specific compute-data ratio and training recipe (e.g., learning rate, data mix, fine-tuning strategy). When data scale, modality mix, or target task deviate from the experimental setup, the optimal architecture may change. Therefore, the paper's conclusions that "pretraining is not necessary and vision-specific capacity is necessary" should be understood as robust trends under controlled conditions, not absolute assertions for all scenarios.

One-line Summary

Over 200 MLLMs trained from scratch prove: vision encoder pretraining is not necessary, but vision-specific parameters cannot be skipped.

Paragraph Summary

A large-scale controlled study by Meta FAIR and NYU trained over 200 MLLMs from scratch and systematically disentangled the role of visual encoding under unified data, compute, and design variables. There are two core conclusions: first, under matched compute, four pretrained encoders at best only tie with encoders trained jointly from scratch—extending MetaCLIP pretraining raised ImageNet linear probing accuracy (80.4% → 84.4%) but did not help VQA; second, encoder-free models lag at every scale and require about 3x the compute to catch up, showing that vision-specific capacity is irreplaceable. Based on this, the authors propose in-fusion, which embeds vision weights into the LLM's early layers, matching a separate encoder while reducing time to first token by about 13%. The study also confirmed the conclusions hold when adding vision capabilities to a pretrained LLM.

#multimodal large language models#vision encoder#Meta FAIR#NYU#in-fusion#VQA#time to first token#MLLM architecture
Share:

Related articles