Multi-Knowledge-Base RAG Architecture: Route First or Retrieve First?

When RAG Systems Face Multiple Knowledge Bases: The Breakdown of Traditional Fusion Strategies

In enterprise RAG (Retrieval-Augmented Generation) architectures, a widely adopted approach is to retrieve top-k results from each data source, then fuse them. When the number of knowledge bases is small, fusion methods like RRF (Reciprocal Rank Fusion) work well.

RAG (Retrieval-Augmented Generation) is an architectural paradigm that combines information retrieval with the generative capabilities of large language models. The core idea is: before the LLM generates an answer, relevant document fragments are first retrieved from external knowledge bases, and these fragments are injected into the prompt as context, enabling the model to generate more accurate, well-grounded responses based on real data. This approach effectively mitigates the LLM "hallucination" problem—where models fabricate plausible-sounding but factually incorrect information—while also breaking through the limitations of the model's training data cutoff date. In enterprise scenarios, RAG systems typically need to interface with multiple data sources—including internal document repositories, product manuals, customer tickets, regulatory databases, and more—which naturally creates a complex multi-knowledge-base architecture.

But when the number of knowledge bases grows beyond 10, does this default strategy still hold? A developer, after deep-diving into enterprise RAG architecture, raised a challenge: Once the number of knowledge bases reaches a certain scale, you're no longer dealing with a simple document ranking problem—you're making implicit comparisons across different retrieval distributions, domains, and corpus sizes.

This fundamentally changes the nature of the problem.

Why Fusion Methods Break Down at Scale

The Core Contradiction: Cross-Corpus Score Incomparability

The key issue lies in cross-corpus score incomparability. Traditional fusion methods carry a dangerous implicit assumption: that similarity scores produced by different corpora are comparable.

In vector retrieval, documents and queries are encoded as high-dimensional vectors, with similarity typically computed via cosine similarity or inner product. However, the vector distribution characteristics of different knowledge bases often vary significantly: specialized domain corpora (such as medical literature) tend to cluster in a specific region of the semantic space, causing retrieval results to have generally higher similarity scores; while general-purpose corpora have more dispersed vector distributions with wider score ranges. Additionally, corpus size directly affects score distribution—the best-matching document in a small knowledge base may have a much higher match score than that of a large knowledge base. This phenomenon is known as the "score calibration" problem in information retrieval, and it represents a fundamental technical barrier to cross-corpus fusion.

Consider this scenario: you have 10 knowledge bases, and each base's top-1 result receives nearly equal weight during fusion. But these bases may differ drastically in corpus size, domain characteristics, and vector distributions:

A small, specialized knowledge base's top-1 result may be highly relevant
A large, noisy knowledge base's top-1 result may simply happen to have a high score

Take RRF as an example. This classic fusion algorithm was proposed by Cormack et al. in 2009, with the core formula: for each document d, the fused score = Σ 1/(k + rank_i(d)), where rank_i(d) is the position of document d in the i-th ranked list, and k is a constant (typically set to 60). RRF's advantage is that it relies only on rank positions rather than raw scores, theoretically circumventing score scale inconsistencies between different retrieval systems. However, when the number of lists being fused increases dramatically, the meaning of rank positions itself becomes ambiguous—the actual relevance of a rank-1 document can vary enormously across corpora of different sizes and quality, which is precisely why RRF fails in multi-knowledge-base scenarios.

If a fixed similarity threshold is used, this assumption becomes even more problematic—it assumes that the score distributions of all corpora are aligned, when in reality they frequently are not.

Good Retrieval Doesn't Equal Good Context

This leads to the following failure chain:

Good retrieval → Problematic cross-corpus ranking → Poor context selection

Each individual knowledge base may have returned the correct results, but during cross-corpus fusion and top-k truncation, the context that's actually needed gets diluted or buried. This is the deep-seated reason why many multi-knowledge-base RAG systems underperform in production environments.

The Architectural Shift: From Retrieve-First to Route-First

Comparing Two Architectural Paradigms

Traditional paradigm (Retrieve-First):

Query → Global Retrieval → Fusion → Hope the right context is in top-k

Routing paradigm (Route-First):

Query → Knowledge Base Routing → Targeted Retrieval → Reranking → Generation

The core insight is: Rather than struggling with cross-corpus comparisons after retrieval, it's better to precisely route the query to relevant knowledge bases before retrieval.

Knowledge Base Routing is essentially a query classification or intent recognition problem. Common implementation approaches include: first, embedding similarity-based routing—generating one or more representative vectors for each knowledge base (e.g., embeddings of base descriptions), and selecting the target base by computing the similarity between the query vector and each base's representative vector at query time; second, LLM-based routing—using a large language model to determine which bases should be queried based on the query content and each knowledge base's description; third, trained classifier-based routing—training a dedicated classification model using historical query logs. Each approach has its tradeoffs: embedding routing is fast but coarse-grained, LLM routing is flexible but high-latency and costly, and classifier routing is precise but requires labeled data and struggles to adapt to newly added knowledge bases.

This approach avoids unreliable score comparisons across heterogeneous corpora, keeping retrieval within genuinely relevant corpora that have relatively consistent distributions.

Limitations of Routing Strategies

Of course, this approach has its weak points:

The router itself can make mistakes. If the classification model routes a query to the wrong knowledge base, no amount of precision in subsequent retrieval can help
Cross-domain questions still require broad retrieval. Some queries naturally span multiple domains, and forcing them into a single base can actually lose critical information

This raises the truly important question: Where is the tradeoff point between routing and global retrieval?

Technical Choices for Production Environments

Facing dozens of knowledge sources and real production traffic, engineers have a full toolkit to choose from:

Major Technical Approaches

Routing/Classification: Determine query attribution first, then perform targeted retrieval. Precise but dependent on routing quality
Global Retrieval + RRF: The classic fusion method—simple but facing the aforementioned issues at scale
Score Normalization: Attempts to solve cross-corpus score incomparability for fairer fusion. Common normalization strategies include min-max normalization (linearly mapping each knowledge base's scores to the [0,1] interval), z-score normalization (standardizing based on mean and standard deviation), and quantile-based normalization. However, normalization itself faces challenges: when a knowledge base contains no truly relevant documents, the highest normalized score can still be pushed up to the same level as relevant results from other bases, creating a "high-score illusion for irrelevant results"
Cross-Encoder Reranking: Using a more powerful model to re-evaluate relevance after initial retrieval, compensating for the coarseness of the first-pass retrieval. Cross-encoders differ fundamentally from the bi-encoders commonly used in the initial retrieval stage—bi-encoders independently encode queries and documents separately, suitable for large-scale initial retrieval but limited in precision; cross-encoders concatenate the query and document and jointly feed them into a Transformer model, achieving deep interaction understanding through attention mechanisms that capture fine-grained semantic relationships between query and document. The cost is significantly higher computation—for N candidate documents, N complete model inference passes are required, so it's typically only used to rerank the top-50 to top-100 results from initial retrieval. In multi-knowledge-base scenarios, a key advantage of cross-encoders is that they directly evaluate query-document pair relevance, unaffected by differences in original retrieval score distributions, providing a relatively unified evaluation baseline for cross-corpus results
Hierarchical Retrieval: Coarse-to-fine, progressively narrowing focus. This architecture draws inspiration from the cascading filtering approach long practiced in traditional search engines. Typical implementations include: the first layer uses lightweight sparse retrieval (such as BM25 keyword matching) or coarse-grained vector retrieval to quickly filter candidate sets from the full corpus; the second layer uses more refined semantic retrieval to further narrow the scope within the candidate set; the third layer uses cross-encoders for fine-grained ranking. In multi-knowledge-base scenarios, hierarchical retrieval can also be extended to a three-level "base-level → document-level → passage-level" structure—first determine relevant knowledge bases, then retrieve relevant documents within selected bases, and finally locate the most relevant passages within documents. This architecture effectively controls computational overhead while maintaining recall
Hybrid Approaches: In practice, it's often a combination of multiple strategies

One-Shot Mode vs. Agentic Multi-Step Retrieval

Some RAG frameworks support both modes:

Agentic Multi-Step Retrieval: The system acts like an agent, progressively converging on the answer through multiple steps. This mode borrows from the AI Agent paradigm—at each step, the model judges whether further retrieval is needed based on currently obtained information, which knowledge base to query, and how to adjust the query strategy. For example, the first step might involve broad retrieval to understand the problem scope, the second step identifies which specialized domain to dive deeper into based on initial results, and the third step performs precise targeted retrieval. The advantage of this approach is its ability to handle complex multi-hop reasoning problems, but the latency and token consumption from multi-step interactions require careful consideration in production environments
One-Shot Mode: The system first selects relevant knowledge bases, then retrieves from them in parallel

This "select bases first, then retrieve in parallel" one-shot mode is essentially an engineering implementation of the routing paradigm—it front-loads knowledge base selection, avoiding blind global retrieval.

Architecture Choices Matter More Than Abstract Wrappers

The value of this discussion isn't about judging which technology's abstraction layer is more elegant. It's about a more fundamental question:

When you have dozens of knowledge sources and real production traffic, what kind of architecture can truly hold up?

Several key judgments deserve deep consideration from every RAG engineer:

With a small number of knowledge bases, fusion methods (RRF) remain a reasonable default choice
At scale, routing/classification is becoming unavoidable, because cross-corpus score comparison is inherently unreliable
Pure routing and pure global retrieval are both extremes. Production-grade systems will most likely need a hybrid architecture that dynamically adapts based on query type—routing for queries with clear domain attribution, falling back to broader retrieval for cross-domain queries

The design philosophy behind this dynamic hybrid architecture is highly consistent with the "Query Understanding" module long practiced in the search engine field. In traditional search engines, the query understanding layer performs intent classification, entity recognition, query rewriting, and other processing on user input, then selects different retrieval strategies and recall channels based on the understanding results. The routing layer in multi-knowledge-base RAG essentially plays the same role—it needs to understand the query's domain attribution and information need type within milliseconds, and make optimal retrieval path decisions accordingly.

Multi-knowledge-base RAG is perhaps truly more of a routing problem: the real challenge isn't "can we retrieve it" but rather "where should we retrieve from" and "how to strike the right balance between accuracy and coverage." This may be the critical watershed for enterprise RAG systems evolving from "functional" to "effective."