Multilingual Retrieval Inconsistency: Diagnosing Hidden Bias in Dense Embedders

An Overlooked Phenomenon: Same Question, Drastically Different Retrieval Results Across Languages

Multilingual Dense Retrievers are designed with a straightforward goal: map documents and queries from all languages into a single vector space, so that no matter what language a user asks in, the system can find semantically relevant documents. It sounds elegant, but a study reveals an unsettling fact — the same question asked in different languages yields dramatically different retrieved documents.

This isn't because the model understands some languages poorly. The study explicitly notes that each language performs quite well when evaluated individually, and different language pairs produce consistent rankings of candidate documents. The problem lies in the "shared index" step: when documents from all languages are mixed into the same index, the correct answers that should rank highly for a given language get pushed far down by a flood of irrelevant documents. These distractor documents are either in the query's own language or in English — depending on the model.

This finding challenges a default assumption in multilingual retrieval: that a shared vector space is inherently good. It turns out that while a shared space is unified, it introduces a "crowding" problem between languages.

Dense Retrieval is a class of methods that contrasts with traditional sparse retrieval (such as BM25): instead of relying on term matching, it uses neural networks to encode queries and documents into low-dimensional dense vectors, then determines relevance through vector similarity (typically cosine similarity or inner product). A multilingual dense retriever further uses a shared encoder (such as multilingual BERT, LaBSE, mE5, etc.) to map text from all languages into the same vector space.

The benefit of this "shared space" is cross-lingual zero-shot transfer: an English query can directly match Chinese documents without building a separate index for each language. But it also carries an implicit assumption — that representations from different languages are evenly distributed in the space and don't interfere with each other. The "crowding" problem revealed by the study is precisely a manifestation of this assumption failing on real data.

Mechanism Analysis: The Problem Is in the Shared Index, Not Language Understanding

The research team conducted a careful investigation into the causes of the aforementioned divergence. The core conclusion: each language maintains a reasonable ranking capability on its own, but the shared index changes the competitive landscape.

Specifically, when the query is in language A, language A's own correct documents do have higher relevance scores, but among the documents in languages B and C, some happen to be closer to the query in vector space — even if they are not semantically relevant. These "neighbor" documents push the truly relevant ones down.

More subtly, the source of distractor documents varies by model. Some models are more easily distracted by irrelevant documents in the query's own language, while others are more susceptible to English documents. This means different models develop different "language preferences" in the shared space — an implicit bias rather than an explicit design choice.

This finding has direct implications for multilingual RAG systems, cross-lingual search engines, and similar applications: users may get retrieval results of vastly different quality simply by switching the language they ask in, and this discrepancy is hard to detect.

A shared index refers to encoding all multilingual documents and storing them in a single vector index (such as FAISS, ScaNN), where retrieval involves nearest-neighbor search on the query vector. The problem is that the vector space of a multilingual encoder is not uniformly spread across languages — it exhibits a language cluster structure: documents in the same language tend to cluster together, while different languages may have partially overlapping regions.

When a query enters this space for ANN search, the returned Top-K candidates will come from multiple languages simultaneously. Since "geometric proximity" in vector space does not always equal "semantic relevance," some documents from other languages that happen to fall near the query will occupy slots. This effect is further amplified as the number of languages increases and the index scale grows, but monolingual evaluation (building an index with only that language's documents) cannot expose it.

Solution: Using "Disagreement" as a Signal to Guide Retrieval

Since the disagreement itself exposes the problem, the research team turned it around and treated disagreement as a usable signal, proposing a disagreement-guided steering method.

The idea is as follows: for a given language, compare "documents that all other languages agree are relevant" with "documents that this language alone retrieves," and take the difference between the two representations as a direction vector specific to that language. At inference time, add this direction to the query vector for that language.

Intuitively: this direction captures "the offset specific to that language in the shared space," and adding it amounts to a language-level correction to the query.

The "direction vector" here draws on the idea of vector arithmetic in representation space: if a language has a systematic offset in the shared space, then the difference between "the average representation of that language's retrieval results" and "the consensus representation across other languages" can approximately characterize this offset direction. Adding it back to the query vector is equivalent to "pushing" the query away from the crowded region and toward the document cluster truly relevant to that language.

This inference-time intervention differs from fine-tuning: the model weights remain completely untouched, and only a single addition is performed on the encoded query vector. It is extremely low-cost and plug-and-play. Similar ideas have been applied in representation engineering and LLM activation steering, and are here transferred to the retrieval setting.

Experimental Comparison: Signal-Guided vs. Other Debiasing Strategies

The study compared the disagreement-guided method against several baselines, including language-mean removal, subspace removal, and per-language rotation.

Results show that disagreement-guided steering is most effective at reducing same-language document crowding, while maintaining competitive recall. However, it does not win across the board — language-mean removal performs better at preserving ranking quality, and there is a trade-off between the two.

This trade-off is itself informative: it shows that "de-crowding" and "preserving ranking" are two objectives that can be optimized separately, and practical systems need to choose strategies based on their scenario. For recall-priority scenarios, disagreement-guided steering is more suitable; for scenarios demanding high ranking precision, mean removal may be the safer choice.

The baselines compared in the paper are all post-hoc debiasing methods, sharing the common trait that they do not retrain the model and only modify representations:

Language-mean removal: Compute the average vector of each language's document representations, and subtract this mean from query or document representations to eliminate language-specific central offsets.
Subspace removal: First perform principal component analysis (PCA) to identify the directions that primarily carry "language identity" rather than "semantics," then project out these subspaces.
Per-language rotation: Learn an orthogonal transformation for each language to align all languages' representations to a unified reference frame.

The key difference of disagreement-guided steering is that it does not use global language statistics, but instead leverages the dynamic signal of disagreement between retrieval results, making it more attuned to the specific question of "which documents were wrongly pushed out in this particular query" — but also more sensitive to the balance between recall and ranking.

Implications for Multilingual Retrieval in Practice

This study serves as a reminder for developers and users of multilingual retrieval systems: the convenience of a shared vector space may come at the cost of hidden cross-lingual interference. When evaluating multilingual retrievers, testing monolingual performance alone is far from sufficient — one must also check whether different languages perform consistently under the same index.

The disagreement-guided method offers a lightweight correction — no model retraining needed, just a direction adjustment to the query vector at inference time. For teams that have already deployed multilingual retrieval systems, this is an optimization direction worth trying.

At the same time, the study leaves open questions: Is this language-specific offset direction stable? Can it transfer across models? And how will this crowding effect evolve as the number of languages in the index increases? These questions are crucial for building truly language-fair retrieval systems.

One-line Summary

Multilingual retrievers' shared index causes cross-lingual interference; the study uses "disagreement-guided" inference-time correction of query vectors.

Paragraph Summary

Multilingual dense retrievers are supposed to map queries and documents from different languages into a unified vector space, but research finds that the same question asked in different languages retrieves significantly different documents. The problem is not language understanding but the shared index: each language ranks reasonably well on its own and consistently with others, but once mixed into a single index, correct documents get pushed out of top ranks by irrelevant documents in the query's own language or in English. The study proposes "disagreement-guided steering," which constructs language-specific direction vectors by comparing the representational difference between a language's solo retrieval results and the cross-language consensus, then adds them to the query at inference time. Experiments show this method best alleviates same-language crowding, though it trades off ranking quality against language-mean removal. This finding reminds developers: multilingual retrieval evaluation cannot rely on monolingual performance alone.