mmVB
A lightweight, plug-and-play visual token compression module for multimodal large language models. Its core idea is Visual Tokens on Demand: it dynamically allocates a different visual token budget to each image based on the query and cross-image context. It adds only about 1.6M trainable parameters and requires no retraining of the base model.
Timeline (last 90 days)
mmVB is a lightweight, plug-and-play module for on-demand allocation of visual tokens that adds only 1.6M trainable parameters, requires no retraining of the base model, and does not need to access the attention matrix at inference time.
The core idea of mmVB is to allocate the visual token budget on demand: based on the query and cross-image context, it dynamically assigns a different token budget to each image, rather than using a uniform compression rate for every image.
On eight multi-image reasoning benchmarks, mmVB reduces visual input tokens by 9×, achieves 5× faster time to first token (TTFT) and 7× fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy.
The 94.4% accuracy retention is an average across eight benchmarks rather than a single-task result, indicating that the benefits of mmVB are general.
The idea of mmVB can be extended to more complex scenarios such as video understanding and multimodal RAG.
All Facts (5)
The core idea of mmVB is to allocate the visual token budget on demand: based on the query and cross-image context, it dynamically assigns a different token budget to each image, rather than using a uniform compression rate for every image.
50%UnverifiedOn eight multi-image reasoning benchmarks, mmVB reduces visual input tokens by 9×, achieves 5× faster time to first token (TTFT) and 7× fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy.
50%UnverifiedThe 94.4% accuracy retention is an average across eight benchmarks rather than a single-task result, indicating that the benefits of mmVB are general.
50%UnverifiedmmVB is a lightweight, plug-and-play module for on-demand allocation of visual tokens that adds only 1.6M trainable parameters, requires no retraining of the base model, and does not need to access the attention matrix at inference time.
50%UnverifiedThe idea of mmVB can be extended to more complex scenarios such as video understanding and multimodal RAG.
50%