Latent-Lens
A cross-model latent-space communication method that projects the hidden states of a frozen vision-language model (VLM) into the embedding space of a frozen small language model (SLM) through lightweight encoders and decoders, enabling multimodal information transfer without textual intermediation. Only about 1.6M parameters are trained.
Timeline (last 90 days)
Latent-Lens is a multi-agent system approach that lets a frozen VLM and a frozen SLM communicate through a latent space.
The core of Latent-Lens is a lightweight codec consisting of a domain encoder and an attention decoder, which trains only about 1.6M parameters, with both backbone models frozen throughout.
The domain encoder projects the VLM's hidden states into the SLM's embedding space, achieving cross-model representation alignment.
The attention decoder is implemented in LoRA form and is used to teach the SLM to attend to the visual prefix.
On the CLEVR dataset of the VQA task, Latent-Lens achieves an accuracy of 81.0±7.9%, versus 58.3±5.7% for the text communication baseline.
On the image captioning task (Flickr8k), Latent-Lens improves by +16.5 over the text baseline with the same LoRA budget and by +18.8 over the frozen version.
The above comparison is conducted under the same LoRA budget, ruling out the explanation that more parameters make it stronger.
Latent-Lens still works on the pair TinyLlama-1.1B and InternVL2-1B, whose architectures, vocabularies and hidden dimensions are all different.
Latent-Lens currently relies on paired training data to learn the projection, and the generalization boundary of the codec and its support for complex visual reasoning tasks still require more validation.
All Facts (9)
Latent-Lens is a multi-agent system approach that lets a frozen VLM and a frozen SLM communicate through a latent space.
50%UnverifiedThe core of Latent-Lens is a lightweight codec consisting of a domain encoder and an attention decoder, which trains only about 1.6M parameters, with both backbone models frozen throughout.
50%UnverifiedThe domain encoder projects the VLM's hidden states into the SLM's embedding space, achieving cross-model representation alignment.
50%UnverifiedThe attention decoder is implemented in LoRA form and is used to teach the SLM to attend to the visual prefix.
50%UnverifiedOn the CLEVR dataset of the VQA task, Latent-Lens achieves an accuracy of 81.0±7.9%, versus 58.3±5.7% for the text communication baseline.
50%UnverifiedOn the image captioning task (Flickr8k), Latent-Lens improves by +16.5 over the text baseline with the same LoRA budget and by +18.8 over the frozen version.
50%UnverifiedThe above comparison is conducted under the same LoRA budget, ruling out the explanation that more parameters make it stronger.
50%UnverifiedLatent-Lens still works on the pair TinyLlama-1.1B and InternVL2-1B, whose architectures, vocabularies and hidden dimensions are all different.
50%UnverifiedLatent-Lens currently relies on paired training data to learn the projection, and the generalization boundary of the codec and its support for complex visual reasoning tasks still require more validation.
50%