BM25 vs Embeddings: Why Can't Vector Search Find Error Code E4012?

Why does vector search miss E4012?

Typing E4012 into a documentation assistant returns pages for E4021 and E4013, while the content that actually explains E4012 ranks 14th—far below the top-5 cutoff. This is a classic conflict between embedding models and BM25 in exact identifier retrieval: semantic vectors excel at understanding "what the error code means" but fail to precisely match "the error code itself".

Embedding models encode the "topic" of a passage, not exact characters. Two pages describing adjacent error codes have nearly identical topics, so their vectors are naturally highly similar. Take E4012 as an example:

• Tokenization: Subword tokenizers split it into E, 40, 12—fragments with nothing distinctive about them; E4021 becomes E, 40, 21, with massive overlap.
• Context mixing: The Transformer fuses each token with surrounding sentences, so descriptions like "connection timeout" around the code dominate the semantics.
• Pooling: Mean pooling compresses hundreds of tokens into a single 384–1024-dimensional vector, where a few numeric tokens have negligible impact on the whole.

As a result, E4012 and E4021 nearly coincide in vector space. The embedding of the short query E4012 is closer to a fuzzy "error code category" vector, with roughly equal distance to every error page—making the ranking almost random.

Subword tokenizers (such as BPE, WordPiece) split text into high-frequency subword units to balance vocabulary coverage with handling of out-of-vocabulary words. For identifiers like E4012, tokenizers typically break them into common fragments like E, 40, 12, diluting exact character information. This design works well for natural language but strips identifiers of their discriminative power in embedding space, because adjacent error codes share many subword fragments and their vector representations are highly similar.

How does BM25 turn things around?

BM25 counts exact term matches and weights them by term rarity across the corpus. Rare terms get extremely high weight, so documents containing them win. Its score has three components:

1. IDF (inverse document frequency): log((N - n + 0.5) / (n + 0.5) + 1). Across ~2,300 text chunks, "connection" appears hundreds of times while e4012 appears only 3 times, giving an IDF of ~6.5—far above "connection"'s 1.5.
2. Term frequency saturation: The k1 parameter (default 1.2–1.5) prevents linear term frequency growth.
3. Length normalization: The b parameter (default 0.75) prevents long documents from winning on length alone.

BM25 doesn't understand what an error code means, but it knows this token is rare and the query contains it—which is exactly the key for identifier lookup.

Basic implementation using rank_bm25:

import re
from rank_bm25 import BM25Okapi

def tokenize(text: str) -> list[str]:
    # Keep E4012, ERR_CONN_RESET, v2.3.1 etc. as single tokens
    return re.findall(r"[A-Za-z0-9_][A-Za-z0-9_.\-]*[A-Za-z0-9_]|[A-Za-z0-9]", text.lower())

corpus_tokens = [tokenize(c.text) for c in chunks]
bm25 = BM25Okapi(corpus_tokens, k1=1.2, b=0.75)

scores = bm25.get_scores(tokenize("E4012"))

IDF (inverse document frequency) measures how rare a term is in the corpus: the fewer documents it appears in, the higher the IDF and the greater its contribution to the score. Across ~2,300 text chunks, e4012 appears only 3 times, its IDF far exceeding common words like connection, so documents containing this rare token score significantly higher in BM25. This is exactly the key mechanism that makes BM25 superior to semantic vectors for exact identifier retrieval.

Tokenizer pitfalls: why did my first BM25 attempt fail?

My first attempt used text.split() directly, causing E4012, (with comma) and E4012 to become different tokens and losing many matches. After switching to a "canonical" tokenizer, v2.3.1 got split into v2, 3, 1, and ERR_CONN_RESET became three common words—rarity completely gone.

Whether you use Elasticsearch, OpenSearch, Postgres full-text search, or rank_bm25, you need to check:

• Does the analyzer split on _, ., -? Test with real identifiers.
• Does stemming break the code? errors → error is fine, but HTTP_403s being mangled is not.
• Do queries and documents use the same tokenizer? Analyzer mismatch is a classic zero-hit bug.
• Lowercasing is usually harmless for code, unless E4012 and e4012 are different things in your product.

The regex above is crude but keeps identifiers intact. Before trusting it, run it against your trickiest IDs.

Hybrid retrieval: fusing BM25 and vectors with RRF

Run both retrievals in parallel, then merge the ranked lists with Reciprocal Rank Fusion (RRF). RRF ignores raw scores and looks only at rank position, because BM25 scores and cosine similarities live on completely different scales.

Formula: for each document, sum its 1/(k + rank) across each list, with k=60 being the paper's common default.

def rrf(result_lists: list[list[str]], k: int = 60, top_n: int = 5) -> list[str]:
    scores: dict[str, float] = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]

dense_ids = vector_search(query, limit=50)
sparse_ids = bm25_search(query, limit=50)
context_ids = rrf([dense_ids, sparse_ids])

For E4012: BM25 ranks the correct page 1st (score 1/61), vector search ranks it 14th (1/74), totaling ~0.030; the E4021 page ranks 1st in vectors but doesn't appear in the BM25 list, scoring only 1/61 ≈ 0.016. The correct page's score nearly doubles. For vague questions like "why is my upload stuck", vector search is stronger, and RRF lets each signal win naturally.

Note: take ~50 candidates from each side, not 5. If the correct document ranks 14th in one list, taking only 10 means fusion will never see it.

Reciprocal Rank Fusion (RRF) is an unsupervised rank fusion method that relies on rank positions in each retrieval list rather than raw scores. For each document, it sums its 1/(k + rank) values across lists, where k is a smoothing constant (commonly 60). RRF effectively balances the contributions of different retrieval signals, preventing any single path from dominating due to score scale differences—ideal for fusing heterogeneous results like BM25 and vector retrieval.

Real-world results on my test set

On my own test set, hybrid retrieval recovered all identifier queries that vector search missed, with natural-language queries essentially unaffected. The numbers below come from my corpus and embedding model—not a universal benchmark.

Setup: 120 real queries drawn from support tickets, each labeled with an answerable text chunk. 38 contain identifiers (error codes, config keys, CLI flags, version strings). Metric: does the correct chunk appear in the top-5.

• Pure vector: 21 of 38 identifier queries hit. Natural-language queries fine.
• Pure BM25 (naive tokenizer): identifier performance worse than expected, for the tokenization-splitting reasons above.
• Pure BM25 (identifier-preserving tokenizer): 36 of 38 hit, but poor on paraphrase questions.
• Hybrid + RRF: all 38 identifier queries hit, with no noticeable loss on natural-language queries at this scale.

120 queries is a small sample and should be treated as "directional" rather than "precise numbers". Building your own eval set from real queries is the only meaningful measurement.

Should a reranker replace this?

Cross-encoder rerankers improve ordering but can only rerank the candidates retrieval returns. If vector search buries the correct chunk at position 14 and you only take the top 10, the reranker never sees it. Rerankers also read the same subword fragments, so they're better than bi-encoders on identifiers but not immune. Put them after hybrid retrieval, not in place of it.

My current low-cost checklist

• Grep your query logs for patterns like [A-Z]+\d{3,}, \w+_\w+, or v\d+\.\d+ to estimate what share of traffic is identifiers. Mine is about one-third.
• For each identifier query, check whether the document containing that exact string makes the top-5. If it doesn't, your retriever is guessing.
• Feed 5 real IDs into your BM25 analyzer and eyeball the tokenization.
• Log both ranked lists before fusion. When an answer is wrong, you can quickly pinpoint which side dropped the ball.

Conclusion: BM25 and embedding models should work together

Embedding models find semantically equivalent passages; BM25 finds passages containing the same rare token. Vector search misses E4012 because embedding models compress code digits into vectors dominated by surrounding semantics, making adjacent error codes nearly overlap; BM25 assigns high IDF weight to rare exact tokens, ranking matching documents first. Run both retrievals in parallel, take ~50 candidates each, fuse with RRF at k=60, and ensure your BM25 tokenizer preserves identifiers intact. That way you can handle both "what does this error mean" and "look up error E4012" queries at the same time.