Allocating Visual Tokens on Demand: A New Approach to Cut Multi-Image Inference Costs by 9x

Why Multi-Image Inference Has Become an Efficiency Bottleneck for Multimodal LLMs

Multimodal large language models (MLLMs) have advanced rapidly over the past two years, yet the vast majority are still optimized around single-image input. When the input becomes multiple images, each image produces a large number of visual tokens, and the encoding overhead and attention computation grow linearly—or even super-linearly—with them.

This problem is extremely common in real-world scenarios. For chart comparison, multi-page document understanding, video-frame reasoning, cross-image retrieval, counting, and temporal-order judgment, the model needs to "see" multiple images simultaneously rather than processing them one by one independently. Redundancy in the single-image setting is mainly spatial and local—for example, adjacent patches within one image are highly similar. In the multi-image setting, redundancy can be cross-image, and task-relevant evidence is often sparsely scattered across different images.

Visual tokens are the basic unit by which multimodal large models process images. A typical approach uses a vision encoder such as ViT to split an image into patches (e.g., 14×14 pixels each); after encoding, each patch becomes a vector that is projected into the language model's word-embedding space, becoming a sequence element isomorphic to text tokens. A 448×448 image with a patch size of 14 yields 1024 visual tokens, and with high resolution or tiling strategies this can reach several thousand. All of these tokens enter the Transformer's attention computation, and their cost grows quadratically with sequence length. In multi-image scenarios, the total number of visual tokens is therefore often several times—or even tens of times—the number of text tokens, becoming the primary source of compute consumption during the prefill stage. Compressing the number of visual tokens is essentially about reducing this redundant computation without breaking semantics.

FlashAttention is an IO-aware exact attention algorithm that fuses the computation of the attention matrix into SRAM through tiling and online softmax, avoiding writing the full N×N attention matrix back to GPU memory. It greatly reduces memory usage and improves speed while remaining numerically exact, and has become the default implementation in mainstream inference frameworks. But precisely because attention scores are consumed on the fly during computation and are not materialized, it is difficult for external modules to directly read the attention matrix to judge which tokens are important. Token pruning or compression methods that rely on attention weights therefore struggle to coexist with FlashAttention—a key reason many training-free compression approaches face engineering constraints in deployment.

Why Existing Compression Methods Fall Short

Around visual token compression, the academic community has already accumulated a number of technical routes: token pruning, token merging, visual token compression, learnable resampling, hierarchical compression, retrieval-augmented visual reasoning, memory-based architectures, and more. Each has its trade-offs, but they share one premise—most are designed for the single-image setting.

More critically, existing methods face two awkward situations in engineering deployment: first, training-free but inference-inefficient, typically requiring reading the attention matrix to judge token importance, whereas memory-efficient implementations like FlashAttention do not conveniently expose it; second, training-dependent, requiring retraining or adapting the base MLLM to a new compressed-token paradigm, which is costly and poorly transferable.

Multi-image reasoning also brings an additional risk: if each image is aggressively compressed, details necessary for comparison, counting, temporal reasoning, retrieval, or evidence localization may be lost. What single-image compression discards is redundancy; what multi-image compression discards may be the answer itself.

Token pruning refers to directly discarding some visual tokens based on some importance score; the discarded information no longer participates in subsequent computation—high compression but irreversible. Token merging, by contrast, aggregates multiple similar tokens into one representative vector, reducing sequence length while preserving information; a common approach is bipartite matching of adjacent patches followed by averaging. Both are training-free inference-time compression. Learnable resampling (e.g., Q-Former, Perceiver Resampler) introduces a set of trainable queries that compress any number of visual tokens into a fixed number of output tokens via cross-attention—stable compression ratio but requires additional training. Hierarchical compression progressively reduces tokens at different depths of the vision encoder, combining compression with feature extraction. These routes perform well on single-image tasks, but their importance judgment is often based on spatial redundancy within a single image, lacking a cross-image global perspective, so directly transferring them to multi-image scenarios easily deletes key evidence by mistake.

mmVB's Core Idea: Giving Each Image Its Visual Budget on Demand

To address these contradictions, researchers propose mmVB—a lightweight, plug-and-play module. Its core idea is Visual Tokens on Demand: rather than applying a uniform compression ratio to every image, it dynamically allocates a different token budget to each image based on the query and cross-image context.

This design has two direct benefits. First, task-irrelevant images can be heavily compressed, yielding budget to images that actually contain evidence. Second, when task-relevant evidence is sparse and scattered, the model can concentrate its "attention resources" on critical images instead of spreading them evenly.

In terms of parameter scale, mmVB adds only 1.6M trainable parameters, negligible relative to the base MLLM. It does not require retraining the base model, nor does it need to access the attention matrix at inference time, making it compatible with efficient implementations such as FlashAttention.

Performance Data: 9x Compression, 5x Speedup, 94.4% Accuracy Retained

The experimental results in the paper are quite convincing. Across eight multi-image reasoning benchmarks, mmVB reduces visual input tokens by 9x, delivering 5x faster time to first token (TTFT) and 7x fewer prefill FLOPs, while retaining 94.4% of full-resolution accuracy.

The significance of these numbers is that they simultaneously improve inference latency and compute cost—the two most core deployment metrics. TTFT directly affects interactive experience, while prefill FLOPs determine server throughput and per-request compute overhead. For applications that need to handle large numbers of multi-image requests—such as multi-page document QA, e-commerce multi-image comparison, and video-frame understanding—optimization at this magnitude is enough to change system architecture choices.

One detail worth noting: the 94.4% accuracy retention is not achieved on a single task, but is the average performance across eight benchmarks, indicating that mmVB's gains are broadly applicable rather than tuned for one type of multi-image task.

TTFT (Time To First Token) refers to the latency from when a request is issued to when the model generates its first output token; it is mainly determined by the prefill stage—the process in which the model performs a forward pass over all input tokens and fills the KV Cache. In multi-image scenarios, the input sequence is greatly lengthened by visual tokens, inflating prefill computation, and TTFT becomes the bottleneck for interactive experience. FLOPs (floating-point operations) is a measure of computation; prefill FLOPs grow roughly linearly with sequence length (quadratically for the attention part), so reducing the number of visual tokens can simultaneously lower TTFT and per-request compute overhead. These two metrics correspond respectively to the response speed perceived by end users and the throughput cost on the server side—the most core optimization goals when deploying multimodal services.

Implications for Multimodal Application Development

The value of mmVB is not just the numbers in a paper. It points to a direction: the efficiency problem of multi-image inference need not be solved only by retraining models or reworking attention mechanisms—it can also be addressed on the input side through a layer of "budget allocation."

For engineering teams, being plug-and-play, not depending on the attention matrix, and adding only 1.6M parameters means it can be integrated into existing inference pipelines at relatively low cost. For researchers, the idea that "cross-image context determines each image's budget" may also extend to more complex scenarios such as video understanding and multimodal RAG.

Multi-image inference is shifting from a niche need to a mainstream one, and how to keep accuracy while driving costs down will be one of the key battlegrounds for multimodal systems in the period ahead.

Multimodal RAG (Retrieval-Augmented Generation) is an architecture that extends retrieval-augmented generation from plain text to mixed image-text corpora: the system first retrieves several images or pages related to the query from a large image-text library, then feeds them together with text into a multimodal large model to generate an answer. Since a single retrieval often recalls multiple images, the input is naturally multi-image, and the ones that actually contain the answer are often just one or two, with the rest being distractors. This aligns closely with the "sparse and scattered evidence" scenario that mmVB targets—on-demand token budget allocation lets truly relevant images obtain high-resolution representations while irrelevant images are heavily compressed, thereby controlling inference cost even when retrieval quality is uneven. Video understanding is analogous: after sampling keyframes from a long video to form a multi-image input, inter-frame redundancy is extremely high, and the budget allocation mechanism can be used to retain keyframes where actions occur while compressing static frames.

One-line Summary

By allocating visual token budgets on demand, mmVB achieves 9x compression and 5x speedup for multi-image inference while retaining 94.4% accuracy.

Paragraph Summary

Multimodal large models face a severe efficiency bottleneck in multi-image input scenarios: each image generates a large number of visual tokens, encoding and attention overhead grow linearly or even super-linearly, and cross-image redundancy plus sparsely scattered task evidence make single-image compression methods hard to transfer directly. Existing approaches are either training-free but inference-inefficient, depending on the attention matrix and thus incompatible with FlashAttention, or require retraining the base model at high cost. Researchers propose the plug-and-play mmVB module, whose core is "allocation on demand": dynamically assigning a different token budget to each image based on the query and cross-image context, heavily compressing task-irrelevant images and shifting budget to key images containing evidence. The module adds only 1.6M trainable parameters, requires no base-model retraining, and does not access the attention matrix. Across eight multi-image reasoning benchmarks, mmVB reduces visual tokens by 9x, speeds up TTFT by 5x, and cuts prefill FLOPs by 7x while retaining 94.4% of full-resolution accuracy, offering a low-cost deployment path for scenarios such as multi-page documents, e-commerce comparison, and video-frame reasoning.