Latent-Lens: Using Latent-Space Communication to Help Small Models Understand Images

Vision-language models (VLMs) and small language models (SLMs) each have their strengths, but when combined into a multi-agent system, they typically can only exchange information through discrete text. Text is a "lossy compression" of visual detail: spatial relationships and subtle textures lose much of their fidelity the moment they are converted into words. For SLMs with limited capacity, this loss is especially damaging—they already depend on the quality of their input signals.

A new study proposes Latent-Lens, which attempts to bypass the text bottleneck: letting a frozen VLM and a frozen SLM communicate directly in latent space.

Where the Ceiling of Text Communication Lies

In modular multi-agent architectures, the VLM is responsible for "seeing," while the SLM handles "reasoning and output." If only discrete tokens are passed between them, the VLM must first compress its visual understanding into natural language, and then the SLM must reconstruct the visual context from text. This process is extremely unfriendly to fine-grained information such as spatial positions and object attributes.

Existing latent-space communication research mainly addresses state transfer between language models, but how a VLM can hand visual information to a separately pretrained, text-only SLM had not been explored before. The reason is not hard to understand: the two models have different architectures, different vocabularies, and different hidden dimensions, so their hidden states cannot be directly connected.

Hidden states refer to the high-dimensional dense vector representations that a model computes internally for input tokens at each layer. They encode information about the current context and serve as the basis for the model's next computation. Unlike the discrete tokens that are ultimately output, hidden states retain continuous, distributed features—for example, the specific semantic role of a word in a given sentence or the spatial relationship of a certain region. If two models share the same architecture and vocabulary, they can directly share hidden states; however, different vocabularies mean that the correspondence between tokens and embedding vectors is inconsistent, and different hidden dimensions make the vector spaces impossible to align directly.

Vocabulary and embedding space: A model maps token IDs to input vectors through an embedding matrix. The vocabulary determines which tokens exist, while the embedding space is shaped by the training process. Even when two independently trained models are given the same piece of text, the resulting embedding vectors lie in different coordinate systems and cannot be directly exchanged. The core difficulty of cross-model latent communication lies precisely here: one needs to learn an invertible, or at least information-preserving, mapping that translates the source model's representations into the target model's embedding space, rather than simply concatenating or averaging them.

How Latent-Lens Works: A Lightweight Codec

The core of Latent-Lens is a lightweight codec placed between two frozen models, with only a very small number of parameters trained. It consists of two parts:

• Domain encoder: projects the VLM's hidden states into the SLM's embedding space, accomplishing cross-model representation alignment;
• Attention decoder: implemented in the form of LoRA, it teaches the SLM to attend to these visual prefixes.

The entire training involves only about 1.6M parameters, while both backbone models remain frozen throughout. This means the method can be directly applied to off-the-shelf open-source models without joint fine-tuning and without damaging the original models' capabilities.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: it freezes the pretrained weights and injects a pair of low-rank matrices as a bypass into the linear layers of a Transformer, using very few parameters to learn task-specific increments. Implementing the attention decoder in LoRA form means that the decoder does not replace the SLM's original structure; instead, it superimposes a trainable low-rank branch onto its attention computation, guiding the SLM to attend to the projected visual prefixes as valid context. This preserves the SLM's frozen state while keeping the trainable parameters at the million scale, making cross-model bridging reproducible on ordinary open-source models.

Experimental Results: Latent Space Clearly Outperforms Text

The study compared the two approaches on visual question answering and image captioning tasks, with text communication under the same LoRA budget as the baseline.

On VQA tasks, Latent-Lens achieved accuracies of 81.0±7.9% and 65.4±5.4% on CLEVR and GQA, respectively, while the text communication baseline reached 58.3±5.7% and 61.8±1.0%. The relative improvement ranges from 30.2 to 69.8 percentage points (relative to frozen text communication).

On the image captioning task (Flickr8k), Latent-Lens improved by +16.5 over the same text baseline and by +18.8 over the frozen version.

Interestingly, these comparisons were conducted under the same LoRA budget, ruling out the explanation that "more parameters make it stronger." The improvement comes from the communication method itself.

CLEVR is a synthetic visual question answering dataset. Images are rendered from simple geometric shapes under controlled lighting and positions, and the questions mainly test counting, spatial relationships, and attribute comparison. The answer space is small and the language is templated, so it is highly sensitive to fine-grained spatial information—the cost of lossy text compression is most evident here. GQA is generated from real scene graphs, with more natural questions and answers and a richer vocabulary, testing compositional reasoning. Flickr8k is an image captioning dataset, with five human-written captions per image, used to evaluate the ability to generate natural language descriptions; compared with multiple-choice VQA, it better tests whether information is preserved in latent communication.

Cross-Architecture Validation

To show that the method is not effective for only one pair of models, the study also evaluated it with TinyLlama-1.1B and InternVL2-1B. These two models differ in architecture, vocabulary, and hidden dimensions, yet Latent-Lens still works. This is important for practical deployment: in real systems, VLMs and SLMs often come from different vendors and different training recipes, so latent-space alignment must be sufficiently general.

What This Means

The value of Latent-Lens lies not in topping some benchmark, but in opening up a new way of combining models: splitting "seeing" and "thinking" across models of different scales without sacrificing visual detail. For edge devices or resource-constrained scenarios, pairing a small SLM with a frozen visual encoder may be more cost-effective than cramming in a complete VLM.

Of course, the current method still relies on paired training data to learn the projection, and the generalization boundaries of the codec and its support for more complex visual reasoning tasks still require further validation. But the idea of "frozen models + lightweight bridge" offers a pragmatic path for multi-model collaboration.

One-line Summary

Latent-Lens uses a lightweight codec to let a frozen VLM and SLM communicate directly in latent space, significantly outperforming text baselines.

Paragraph Summary

This article introduces a new study, Latent-Lens, which aims to solve the problem that when a vision-language model (VLM) is combined with a small language model (SLM), information can only be passed through discrete text. As a "lossy compression" of visual detail, text loses spatial relationships and subtle textures, which is especially disadvantageous for SLMs with limited capacity. Latent-Lens inserts a lightweight codec of only about 1.6M parameters between two frozen models, including a domain encoder and an attention decoder implemented as LoRA, projecting the VLM's hidden states into the SLM's embedding space and teaching the SLM to attend to visual prefixes. On CLEVR, GQA, and Flickr8k, latent-space communication clearly outperforms text baselines under the same LoRA budget, and its generality is validated on heterogeneous models such as TinyLlama-1.1B and InternVL2-1B.