[KongchangAI]
· 2 min read· 1,156 words

200+ Model Experiments: Vision Encoder Pretraining Not Required, but Vision-Specific Parameters Are Indispensable

200+ Model Experiments: Vision Encoder Pretraining Not Required, but Vision-Specific Parameters Are Indispensable

200+ controlled experiments show vision needs dedicated parameters—but not separate pretraining or an independent encoder.

Meta FAIR and NYU researchers ran controlled experiments with 200+ MLLMs trained from scratch, decomposing the "standard vision encoder" into three independently testable variables: separate pretraining, vision-specific parameters, and placement before the LLM. Under a unified lifecycle compute budget, the conclusions are clear: separate encoder pretraining at best ties joint training from scratch, and extending MetaCLIP pretraining improves ImageNet accuracy without improving VQA; vision-specific parameters are a hard requirement, with encoder-free models lagging at every scale and needing ~3× compute to catch up; the In-fusion structure adds zero-initialized vision-specific weights to the first 8 LLM layers, matching standard encoder performance while cutting time-to-first-token by ~13%. Vision-specific pathways work best in shallow layers.

Multimodal large models (MLLMs) have long lacked a unified answer when it comes to vision encoder design: Gemma 4 12B chose to drop the encoder, Kimi K3 trained the encoder from scratch, and Inkling 2 Small added the encoder back. Published models differ in data, compute, and encoder pretraining setups, making it hard to judge which choice actually works. Researchers from Meta FAIR and NYU ran a controlled experiment with over 200 models trained from scratch, decomposing the "standard encoder" into three independently testable choices: whether to pretrain it separately, whether to keep vision-specific parameters, and whether to place it before the LLM.

Reddit discussion thread screenshot

The core conclusion of the paper "Unbundling Multimodal Vision Encoding" can be summed up in one sentence: vision needs its own parameters, but it doesn't need its own pretraining, nor does it need a separate encoder module.

Experimental Design: 200+ Models, Unified Budget, Trained from Scratch

The research team trained over 200 MLLMs, with the LLM portion also trained entirely from scratch, and all models using the same data. The key design was to count encoder pretraining within the same "lifecycle compute budget"—compute consumed by pretraining the encoder eats into the compute available for multimodal training, ensuring a fair comparison.

The main experiments were based on Qwen3-1.7B and 4B configurations, with visual capacity fixed at 0.4B parameters, plus a scale ladder from 0.6B to 6.4B backbone models. Evaluation covered Cambrian's 16 VQA benchmarks (448 pixels) and 12 language benchmarks, the latter examining common sense, question answering, and broad knowledge. This approach of evaluating vision and language capabilities separately makes it clear which side a design choice actually improves.

"Lifecycle compute budget" is key to understanding the fairness of the experiments in this paper. In conventional practice, vision encoder pretraining is often treated as an "extra cost" independent of multimodal training—an encoder pretrained on billions of image-text pairs doesn't have its compute counted in the subsequent multimodal alignment stage. But this distorts comparisons between different approaches: models with pretrained encoders actually use more total compute. This paper counts all compute (including encoder pretraining) into the same budget pool—spend more on pretraining, and less is left for multimodal joint training. This forces each design to compete under equal total compute, making conclusions comparable. This setup also explains why "free pretraining" isn't free—extending pretraining necessarily eats into downstream training resources.

Finding 1: Separately Pretraining the Encoder Is Optional

The researchers pretrained encoders with four objectives: contrastive learning, self-supervision, and two types of captioning. Under equal total compute, these pretrained encoders at best tie with "training the encoder from scratch alongside the LLM."

More counterintuitively, "free pretraining" doesn't help either. The team extended MetaCLIP's pretraining time to 6.5×, raising ImageNet linear probe accuracy from 80.4% to 84.4%, but VQA performance didn't improve accordingly. This suggests that progress in visual representations on classification tasks doesn't necessarily transfer to multimodal question answering that requires alignment with language.

The engineering implication is: if there's no existing encoder matching the target data distribution, joint training from scratch is a reasonable choice, and there's no need to compromise merely to save pretraining compute.

ImageNet linear probe is a classic method for evaluating visual representation quality: freeze the encoder and train only a linear classifier on its output features—higher accuracy means features are more linearly separable. This paper uses it to test whether pretraining makes the encoder "learn better visual representations." Results show that even with probe accuracy rising from 80.4% to 84.4%, VQA performance remains unchanged. The reason is that VQA doesn't need mere object classification ability, but rather the alignability of visual features with language semantics. Classification tasks optimize inter-class decision boundaries, while multimodal question answering requires visual tokens to be effectively "read" by the LLM and participate in reasoning. The two objectives don't fully overlap, so improvements in classification metrics can't serve as a proxy for multimodal capability.

Finding 2: Vision-Specific Parameters Are a Hard Requirement

This is the strongest conclusion from the experiments. Encoder-free models lag at every scale from 0.6B to 6.4B, requiring about 3× the compute to catch up. Giving the encoder's parameter count directly to the LLM improves text capability, not VQA.

In other words, visual tokens must have their own dedicated parameter pathway. The parameter count itself isn't the bottleneck—the bottleneck is whether these parameters are shared with text.

Finding 3: Placing the Encoder Before the LLM Is Also Not Required

The researchers proposed a structure called "In-fusion": adding zero-initialized vision-specific weights (attention projections and FFN) to the first 8 layers of the LLM, with text tokens never using these weights. From 1.7B to 6.4B, it performs comparably to the standard encoder while reducing time-to-first-token by about 13%.

Ablation experiments give a more detailed picture: at 1.7B, placing vision-specific branches in layers 1–8 achieves 45.0 VQA (vs. 44.4 for the encoder approach), while sharing these parameters with text yields only 42.3, and placing the branches in the last 8 layers drops to 35.1. Vision-specific pathways work best in shallow layers. This conclusion also holds in mid-training scenarios with pretrained LLMs—when adding vision capability to a pretrained 1.7B LLM, the pattern remains unchanged.

"Zero initialization" is the key trick that makes the In-fusion structure train stably. Setting the initial values of vision-specific attention projections and FFN weights to zero means that at the start of training these branches have no effect on the LLM's output, making the model equivalent to a pure-text LLM. As training progresses, visual signals gradually "activate" these branches through gradients, avoiding the representational disturbance caused by random initialization. This follows the same idea as zero-initializing the B matrix in LoRA: first keep the original model's behavior unchanged, then gradually introduce new capabilities. Zero initialization also brings another benefit—text tokens never pass through these vision-specific weights, fundamentally ensuring the purity of the text pathway, and explaining why VQA drops when parameters are shared with text: sharing forces the same set of weights to serve both modalities simultaneously, creating capability competition.

Implications for Model Selection

Some of the current divergence across teams on encoders is actually confusion caused by "bundled choices": removing the encoder often also removes vision-specific parameters, so the model gets worse, but the cause is attributed to "not having an encoder." After separating the three variables, this research points to a simpler design space—

  • You can omit the separate vision encoder and its pretraining;
  • You cannot omit parameter capacity dedicated to visual tokens;
  • The vision pathway doesn't have to be before the LLM; fusing it into shallow layers works equally well and yields lower inference latency.

The author noted their authorship in the post and expressed willingness to answer questions. For teams designing multimodal architectures or evaluating open-source MLLMs, this set of experiments offers a more accurate way to frame the question than "encoder or no encoder."

References: paper (alphaxiv), project blog and FAQ (unbundling.github.io), author's thread on X.

#MLLM#vision encoder#multimodal large models#encoder pretraining#In-fusion#VQA benchmarks#Meta FAIR
Share:

Related articles