Multimodal Indexes Can Reconstruct Original Images: The Privacy Blind Spot in Vector Databases

Visual document retrieval systems slice each scanned page into thousands of patch vectors and store them in third-party vector databases. For a long time, engineers have assumed these dense multi-vector indexes are "low-risk artifacts"—after all, only raw pixels are considered sensitive data. But a new study overturns this assumption: using the index alone, an attacker can reconstruct a page that closely resembles the original, and even precisely identify its source document.

The Underestimated Index: From "Harmless Embeddings" to Privacy Liability

Visual document retrieval pipelines such as ViDoRe typically export per-patch embeddings as a whole and host them in third-party vector databases. The common logic among engineering teams is: vectors are high-dimensional floating-point numbers that, detached from the original model and page geometry, cannot be reverted into readable content, and therefore pose no leakage risk.

This study directly disproves that. The authors built an inversion pipeline that remaps multi-vector indexes back into pixel space. The reconstructed pages can recover about 47% of the original text on the ViDoRe v3 benchmark, and rank their source document first in 98.4% of queries. In other words, an attacker doesn't need access to the original image—just the index—to simultaneously achieve content reconstruction and source identification.

For any commercial retrieval service that outsources multi-vector indexes to external storage, this means the index itself is data that must be treated as a confidential asset.

ViDoRe is a visual document retrieval benchmark proposed by Illuin Tech, short for Visual Document Retrieval, specifically designed to evaluate models' retrieval capabilities on image-based documents such as scans, slides, and posters. It adopts a ColBERT-style late interaction mechanism: a page is sliced by a visual encoder into patches, each generating a dense vector; queries are likewise encoded as multi-vectors, and scoring is done via per-vector max-similarity summation such as MaxSim. As a result, a single page leaves thousands of patch vectors in the index. ViDoRe offers multiple versions including v1, v2, and v3, covering different languages and document types, and is often used as a public evaluation standard for multimodal retrievers. Understanding this pipeline is the core premise of this article: when the index exists in a "per-patch multi-vector" form, the vectors themselves carry far more spatial and semantic information than a single whole-page embedding.

Embedding inversion refers to attack techniques that reconstruct the original input from dense vectors. Early work targeted image classification models by training a decoder or directly optimizing pixels so that the reconstructed image's vector aligns with the target vector in embedding space. For text embeddings, similar methods recover the original text token by token. Such attacks work because the encoder is a deterministic mapping: as long as the attacker has the same or sufficiently similar encoder, "finding the original image" can be transformed into an optimizable inverse problem. Patch-level multi-vector indexes are especially fragile—each vector corresponds to a local region, and the attacker can invert them one by one and reassemble them according to geometric layout, thereby bypassing the blurring caused by the global information compression of whole-page embeddings. This is precisely the mechanism behind the paper's claim of recovering about 47% of the text.

Defenses: Simple and Effective, but with Illusions

Researchers tested two common defenses: token pooling and vector order shuffling. Both can suppress text recall to about 8%, giving the surface impression that leakage has been largely mitigated.

But shuffling-based protection is fragile. The paper points out that a model specifically trained to recover the order can boost the proportion of "source page ranked first" from 3.8% to 93.5%—almost back to the undefended level. In other words, order obfuscation provides only an illusion of security; as long as the attacker is willing to invest in a medium-scale learning model, the shuffled structure can be reassembled.

By contrast, although the pooled version can still be inverted, text recall remains low, and the actual leakage risk is clearly lower. This suggests that defense design should prioritize destroying patch-level semantic granularity, rather than merely hiding the arrangement.

Token pooling in this context refers to aggregating multiple patch vectors within a page into fewer, coarser-grained vectors, such as mean pooling, max pooling, or merging by fixed windows. The goal is to reduce the spatial resolution of vectors in the index, so that a single vector no longer corresponds to a small pixel region, thereby weakening local information during inversion. Vector order shuffling, by contrast, keeps vector contents unchanged and only randomizes their arrangement in the index, attempting to prevent attackers from reconstructing the spatial layout of patches. The former destroys semantic granularity; the latter only hides structural relationships. The experimental results in this paper precisely reflect this essential difference: pooling keeps recall stably low, while shuffling only raises the attacker's reconstruction cost, and once supplemented with an order-recovery model, it nearly fails.

Boundaries of the Threat Model and Unresolved Questions

The paper's threat model has two premises: the attacker possesses exactly the same vision-language encoder and knows the page's geometric layout. The experiments are also concentrated on two retriever types and do not cover the entire multimodal ecosystem. This means there may be additional attack paths in the real world, as well as mitigation measures not touched upon in the paper.

More critically, the paper explicitly states: defenses that rely solely on obfuscating vector order can be reversed by a moderately sized learning model. Privacy protection cannot be built on "cheap tricks." If deployers stake security on shuffling or lightweight obfuscation, the defense line will quickly collapse once attackers upgrade their methods.

Direct Implications for Engineering Practice

The conclusions of this work have several direct implications for retrieval system architecture:

• The index is sensitive data: Multi-vector indexes outsourced to third-party vector databases must be stored encrypted just like the original text, especially at rest.
• Evaluation must include inversion tests: In model and pipeline evaluation, "anti-inversion capability" should be a formal metric, rather than testing only recall and latency.
• Defense strategies must be layered: The difference in effectiveness between pooling and shuffling shows that destroying patch-level semantics is more valuable than hiding order; combining the two still requires careful evaluation.

As multimodal RAG and visual document retrieval rapidly spread in enterprise scenarios, index leakage may become an overlooked data exfiltration channel. Before handing embeddings to external services, teams need to answer a fundamental question again: can these vectors actually be reconstructed?