Seven Evolutions of Retrieval Systems: The Complete Technical Lineage from BM25 to Generative Retrieval

Introduction: A Mental Model for Retrieval Systems

Information Retrieval and Ranking systems form the technical backbone of search engines, recommendation systems, and today's RAG (Retrieval-Augmented Generation) architectures. However, in practice, many developers fall into a common misconception: assuming that new technologies always replace old ones.

Recently, a tech blogger shared a mind map on Reddit about the evolution of retrieval systems, accompanied by a detailed article. He proposed a clear evolutionary lineage, dividing the development of retrieval technology into seven stages (or "waves"). The value of this framework lies not only in organizing history, but in revealing a key insight: these methods don't replace each other — they stack on top of one another and work in concert.

A Panoramic View of the Seven Waves

The evolutionary path proposed by the author is as follows:

Lexical Retrieval → Collaborative/Behavioral Retrieval → Learned Sparse Retrieval → Dense/Two-Tower → Hybrid Retrieval → Multi-Vector/Multimodal → Generative/Agentic Retrieval

Let's break down the technical substance and context of each stage.

Wave 1: Lexical Retrieval — BM25 and TF-IDF

The most classic representatives are BM25 and TF-IDF. These methods are based on literal keyword matching and term frequency statistics. The core idea is: "The more frequently and exclusively a query term appears in a document, the higher the relevance."

The strength of lexical retrieval lies in its strong interpretability, computational efficiency, and zero training requirement — it remains the default baseline in many production systems to this day. Its weakness is equally obvious: it cannot understand semantics. "Car" and "automobile" don't match literally, so they'd be deemed irrelevant — this is the well-known "vocabulary gap" problem.

Wave 2: Collaborative/Behavioral Retrieval

This stage introduced user behavioral signals — clicks, purchases, dwell time, and more. Collaborative filtering is the core idea behind recommendation systems: "People who liked A also liked B."

The value of behavioral signals is that they capture "collective wisdom" that the text content itself cannot express. But it also suffers from the cold-start problem: performance degrades when new items or new users lack interaction data.

Wave 3: Learned Sparse Retrieval — SPLADE and DeepImpact

Learned sparse retrieval, represented by SPLADE and DeepImpact, attempts to combine the efficiency of lexical retrieval with the semantic understanding of neural networks. It still produces sparse vectors (enabling acceleration via inverted indexes), but each term's weight is learned by a neural network, and it can perform "term expansion" — augmenting documents with semantically related terms that don't appear literally in the text.

This stage can be seen as a bridge between traditional retrieval and deep learning.

From Dense Representations to Hybrid Fusion

Wave 4: Dense/Two-Tower Retrieval

With the maturation of pre-trained language models like BERT, dense retrieval became mainstream. The Two-Tower architecture uses two separate encoders to map queries and documents into the same low-dimensional vector space, measuring relevance through vector similarity (e.g., cosine similarity).

Combined with ANN (Approximate Nearest Neighbor) indexing techniques (such as HNSW and FAISS), dense retrieval can achieve millisecond-level recall across millions or even billions of vectors. It truly solves the semantic matching problem, but at the cost of reduced interpretability and high sensitivity to training data quality.

Wave 5: Hybrid Retrieval — BM25 and Dense Retrieval Working Together

Practice quickly proved that dense retrieval is not a silver bullet. It actually underperforms BM25 when handling proper nouns, code, numbers, and other exact-match scenarios. Thus hybrid retrieval emerged — running lexical and dense retrieval simultaneously, then merging results through fusion strategies (such as RRF, Reciprocal Rank Fusion).

This is precisely the core point the author emphasizes: a mature production system often simultaneously incorporates BM25, dense retrieval, ANN, hybrid fusion, behavioral signals, and query rewriting. These are not replacements for each other, but components each fulfilling their own role.

Toward the Frontier of Multimodal and Generative Retrieval

Wave 6: Multi-Vector/Multimodal Retrieval

Representing an entire document with a single vector loses fine-grained information. Multi-vector retrieval, represented by ColBERT, retains an independent vector for each token in the document and achieves more granular matching through a "late interaction" mechanism, significantly outperforming single-vector approaches in accuracy.

On another front, multimodal retrieval breaks beyond the boundaries of text — images, audio, and video can all be encoded into a unified semantic space, enabling cross-modal retrieval capabilities such as "text-to-image search" and "image-to-image search."

Wave 7: Generative/Agentic Retrieval

The latest wave integrates large language models directly into the retrieval pipeline. Generative retrieval lets the model directly generate document identifiers or answers, while Agentic Retrieval endows the system with autonomous planning capabilities: the model can independently decompose queries, perform multi-turn retrieval, evaluate result quality, and decide whether supplementary searches are needed.

This stage deeply converges with the currently hot RAG and AI Agent architectures, representing a paradigm shift in retrieval systems from "passive response" to "active reasoning."

Core Insight: Stacking, Not Replacing

The most valuable aspect of this mind map isn't that it lists seven technical terms, but that it conveys an engineering philosophy: the evolution of retrieval technology is cumulative.

New methods never emerge to retire old ones — they emerge to fill gaps that old methods cannot cover. When you build a real production-grade retrieval system, you'll likely see BM25 handling exact matches, dense vectors handling semantic generalization, behavioral signals introducing personalization, query rewriting optimizing recall, and generative modules providing final reasoning — all these technologies working together on the same pipeline.

Understanding this helps engineers avoid the trap of "blindly chasing the new" and instead choose the right combination of technologies based on the specific needs of their business scenarios. For teams currently building RAG systems, this evolutionary framework also serves as a practical technology selection roadmap.

Note: This article is based on a single source shared in the Reddit community. The author's full article is still being written; interested readers can follow their Substack for future updates.