RAG Hybrid Search: Why Vector Retrieval Misses TS-999 and a Simple Fix

A user types "How do I fix error TS-999?", your RAG app runs a vector search, and pulls back five passages about "generic error codes". The model then writes a confident but useless answer—the one document that actually contains "TS-999" never made it into the prompt.

This isn't a model problem, it's a retrieval problem. And fixing it doesn't require switching to a bigger model or tuning parameters—just a simple, reliable approach: Hybrid Search.

Why Pure Vector Search Misses Exact Matches

Embeddings are great at capturing semantics. "Reset password" and "I can't log in" sit close together in vector space, which is exactly what we want. But an embedding essentially compresses text into a single point, and rare, precise tokens get blurred out during that compression.

The following kinds of content are most likely to be lost in pure vector retrieval:

• Error codes and ticket IDs (TS-999, INC-4821)
• SKUs, part numbers, invoice numbers
• Function names and config keys (max_retries, PG_POOL_SIZE)
• People names, product names, and internal tool names

To an embedding model, TS-999 and TS-998 look almost identical; but to a user, they're two completely different problems. Anthropic uses exactly this example in their official Contextual Retrieval article: embedding models "may find generic content about error codes but miss the exact 'TS-999' match." Keyword search, meanwhile, hits it instantly.

The Core Mechanism of BM25

BM25 (Best Matching 25) is a probabilistic retrieval function introduced in the Okapi system, and it remains the default similarity algorithm in Lucene/Elasticsearch to this day. It computes three factors for each query term: term frequency (TF), inverse document frequency (IDF), and document length normalization. IDF gives rare terms like TS-999—which appear only a few times across the entire corpus—extremely high weight, while high-frequency terms like the or of get weights approaching zero. The TF component uses a saturation function to avoid linearly amplifying the score 100x when a term repeats 100 times; document length normalization prevents long documents from dominating just because they contain more words. It's this combination of IDF and exact token matching that lets BM25 lock onto the chunk containing the literal TS-999 within milliseconds—precisely the information lost after embedding compression.

The Fix: Dual Retrieval, Then Fused Ranking

Hybrid search runs two retrievers on every query:

1. BM25 (keyword search): scores by exact term matching, with higher weight for rare terms. It's the classic search engine ranking function, highly sensitive to IDs and terminology.
2. Vector search (semantic): scores by semantic similarity, strong at handling paraphrases, synonyms, and fuzzy questions.

Then you merge the two ranked lists. The simplest effective fusion method is Reciprocal Rank Fusion (RRF):

score(chunk) = Σ 1 / (k + rank_in_list)

If a chunk appears in multiple lists, you sum its score from each list. k is a constant, typically 60, from the original RRF paper by Cormack, Clarke, and Büttcher at SIGIR 2009. Elasticsearch also uses the same default as its rank_constant in its RRF documentation.

The beauty of RRF is that it only looks at rank, not raw scores. You don't have to agonize over how to compare a BM25 score of 14.2 with a cosine similarity of 0.81.

Why RRF Is Immune to Score Scale

BM25 outputs unbounded real numbers (possibly 0 to 30+), cosine similarity is fixed between -1 and 1, and the two can't be directly summed with weights. RRF sidesteps this: it takes only the rank rank from each list and converts it into a score using 1/(k+rank). A chunk that ranks 1st in BM25 and 40th in vector retrieval can still land high thanks to the sum of its ranks across both paths. The constant k=60 serves as smoothing—it compresses the gap between 1st and 2nd place, making "appears in multiple paths" more important than "first in a single path." This also means RRF doesn't require both retrievers to return the same number of results, making it very easy to implement in practice.

A Minimal Working Python Implementation

The code below uses rank_bm25 for keyword search and reuses your existing implementation for the vector search part:

from rank_bm25 import BM25Okapi

chunks = load_chunks()                      # list[str]
bm25 = BM25Okapi([c.lower().split() for c in chunks])

def keyword_search(query: str, n: int = 50) -> list[int]:
    scores = bm25.get_scores(query.lower().split())
    return sorted(range(len(chunks)), key=lambda i: scores[i], reverse=True)[:n]

def vector_search(query: str, n: int = 50) -> list[int]:
    ...  # your existing embedding retrieval, returns chunk indices by rank

def rrf(*ranked_lists: list[int], k: int = 60) -> list[int]:
    fused: dict[int, float] = {}
    for ranked in ranked_lists:
        for rank, idx in enumerate(ranked, start=1):
            fused[idx] = fused.get(idx, 0.0) + 1.0 / (k + rank)
    return sorted(fused, key=fused.get, reverse=True)

def hybrid_search(query: str, top_k: int = 10) -> list[str]:
    ids = rrf(keyword_search(query), vector_search(query))
    return [chunks[i] for i in ids[:top_k]]

In production you usually don't need to hand-roll BM25. Postgres full-text search, Elasticsearch/OpenSearch, and vector databases like Weaviate all have built-in hybrid search capabilities with the same underlying logic.

How Big Is the Improvement?

Anthropic's experiments across three corpora—codebases, fiction, and papers—showed that combining embeddings with BM25 outperforms embeddings alone; and if you also add context to each chunk and pair it with BM25, top-20 retrieval failure rate drops by 49% (from 5.7% to 2.9%). Adding a reranker on top of that brings the failure rate reduction to 67% (from 5.7% to 1.9%). Source: Anthropic's "Introducing Contextual Retrieval."

Your specific numbers will differ, which is exactly why the evaluation step below is non-negotiable.

Contextual Retrieval and Rerankers

Anthropic's Contextual Retrieval means that before indexing each chunk, you first use an LLM to generate a contextual prefix describing where that chunk sits within the whole document, then embed and BM25-index it together with the chunk. This solves the problem of lost referents after chunking (e.g., a passage that only says "this parameter defaults to 60" becomes retrievable once you prepend "the k value of RRF"). A reranker is a second-stage model, usually based on a cross-encoder, that scores query and candidate chunks together pair by pair; it's more accurate than dual-tower embeddings but computationally expensive, so it only reranks the top 50–150 retrieved candidates. Together, these two explain the difference between the 49% and 67% failure rate reductions in Anthropic's data.

Implementation Roadmap

1. Collect 30–50 real queries from your logs, making sure they include IDs, codes, or exact names.
2. Label the correct chunk for each query.
3. Measure recall@10 for pure vector search as a baseline.
4. Re-measure after adding BM25 + RRF, paying special attention to queries containing IDs.
5. Optionally add a reranker (retrieve ~50–150 first, then keep the top 10–20 after reranking).
6. Keep the evaluation set in your code repository to prevent the next "tiny retrieval tweak" from silently regressing performance.

When You Can Skip It

Hybrid search isn't a silver bullet. If all your queries are fuzzy natural-language questions with no IDs or terminology, pure vector retrieval may be enough. Or if your knowledge base is small enough to fit directly into the prompt (Anthropic suggests trying direct insertion for anything under 200K tokens).

Otherwise, as long as users might enter codes, names, or numbers, you should add keyword search back in.

Key Takeaways

• Vector search is great at semantics, poor at exact tokens.
• BM25 is great at exact tokens, poor at semantics.
• Run both, fuse with RRF (k=60), and measure recall on real queries.

Is your retrieval stack currently pure vector or hybrid? Share your real-world results in the comments.