Why Your 0.8 Similarity Threshold Breaks After Switching Embedding Models

A RAG bot had one safety rule: if the cosine similarity of the best document chunk fell below 0.8, answer "I don't know." That threshold held steady for months — until the author swapped in a new embedding model and redeployed. Suddenly the bot answered "I don't know" to everything, including "how do I reset my password," a question with its own dedicated documentation page.

No crash. No error. The retrieval still returned the correct chunk in first place — the "safety gate" just threw it out every time. Worse, digging deeper revealed the old system was broken too, just in the opposite direction.

What Actually Happened After the Model Switch

After the migration, ranking held up fine, but the score scale collapsed entirely. All relevant matches landed well below the 0.8 threshold, so the gate rejected every result.

Here's roughly what the author logged on the same internal documentation corpus:

Metric	Old model (ada-002)	New model (text-embedding-3-small)
Top-1 score for questions with real answers	Median ~0.84	Median ~0.55
Score between two random, unrelated chunks	Median ~0.72	Median ~0.18

Look at the right column first: the new model is actually better at distinguishing relevant from irrelevant content — the gap between "real answer" and "noise" is much wider. The threshold just happened to sit above the entire distribution.

Now look at the left column: with the old model, random garbage scores 0.72 and real answers score 0.84. That "safe" 0.8 threshold was wedged in a gap of just 0.12 between garbage and gold. The author ran an off-topic test question about Neapolitan pizza against a Kubernetes operations manual and got a score of 0.77 — the guardrail was holding on by a margin of three hundredths.

In other words, that threshold was never a guardrail. It was a coincidence that happened to work for one particular model.

Why Unrelated Texts Score So High on Cosine Similarity

The root cause is that most embedding models are anisotropic: their vectors don't spread uniformly across the hypersphere — they cluster inside a narrow cone. Any two vectors within that cone form a small angle, so cosine similarity stays high regardless of what the text actually says.

This isn't unique to any one vendor. It was documented as far back as studies of BERT and GPT-2 representations (Ethayarajh, 2019 found that contextual embeddings occupy a narrow cone), and "representation degeneration" has been studied in language models for years.

The math takes ten seconds to follow. Model each embedding as a shared component plus a unique component:

v = m + r
|m|^2 = a^2   (direction shared by all texts)
|r|^2 = b^2   (what makes this text unique)

If the unique components are approximately orthogonal to each other and to m, the cosine between two unrelated texts is:

cos(v1, v2) ≈ a^2 / (a^2 + b^2)

If the shared direction accounts for 70% of each vector's squared magnitude, two completely unrelated texts score around 0.7 — real semantics get crammed into the remaining 0.3. That's exactly what ada-002's numbers show. The ratio a^2 / (a^2 + b^2) varies across models — this ratio is your noise floor, and it doesn't ship on the label.

Anisotropy refers to the uneven distribution of directions in a vector space. Ideally, an embedding model should map texts with different meanings to directions spread uniformly across a high-dimensional sphere, so that any two unrelated texts have an expected cosine similarity near zero. Real language models don't work that way. Because of statistical regularities in language — high-frequency words, syntactic structure, topic distributions — all text embeddings shift toward a few "common directions," forming a narrow cone. This means even two completely unrelated texts point in roughly the same direction, their angle is small, and cosine similarity is naturally inflated.

This problem existed in the era of static word vectors (GloVe vectors are particularly anisotropic) and persisted into the Transformer era, where contextual modeling reduced but never eliminated it. Training data scale, training objectives, and post-processing all affect the degree of anisotropy in the final model — which is why two seemingly similar embedding models can have very different noise floors.

Why Score Scales Shift With Each Model

Because the training objective never asked for any specific score. Most embedding models are trained with contrastive loss (InfoNCE or a variant), which only cares whether the correct pair beats the incorrect pairs.

The loss for a single query looks like:

loss = -log( exp(s_pos / τ) / Σ_i exp(s_i / τ) )

Adding the same constant c to every similarity s_i doesn't change the softmax — the exp(c / τ) factor cancels in numerator and denominator. So the model is free to place "unrelated" at 0.1 or at 0.7; training doesn't care.

The temperature τ and training data determine how spread out the scores end up. Switch models and you've switched the recipe and the unit system — your 0.8 won't survive the migration.

There's another trap: if you truncate embeddings yourself (cutting a Matryoshka-style vector to fewer dimensions), the vectors are no longer unit length. Always re-normalize after truncation, or what you're calling "cosine" is actually a dot product with drifting norms.

Contrastive loss, especially InfoNCE (Info Noise-Contrastive Estimation), is the core training objective of most mainstream text embedding models. The basic idea: given a query, the model must identify the true matching positive sample from a batch of candidates while pushing down scores for all negatives. This is essentially a multi-class classification problem — the loss function only cares about the positive sample's relative rank, not the absolute value of any score.

Batch size and negative sampling strategy have a large influence on the final score distribution. When training with in-batch negatives, the model never sees what the absolute score gap between a hard negative and a positive should be — it only learns "the positive must score higher than everything else in this batch." This means scores across different models aren't comparable in absolute terms. Some models compress the positive/negative contrast into the range 0.7–0.9; others spread it across 0.1–0.9. It all depends on the training recipe, and those details rarely appear in model documentation.

Can You Use Cosine Similarity Thresholds at All?

Yes — but only if you've measured the threshold on your corpus with your exact model, and re-measured it every time either one changes. A threshold copied from a blog post, a tutorial default, or a previous model is, in essence, a guess.

The ranking that vector retrieval gives you is generally trustworthy. The absolute number next to it is not — until you calibrate it.

How to Calibrate a Cosine Similarity Threshold

Two steps: measure the noise floor first, then fit the threshold on labeled examples.

Step 1: Noise floor. Randomly sample chunk pairs from your own index and look at their distribution. This takes less than a minute and tells you what "unrelated" means for this specific model:

import numpy as np

def noise_floor(E, n=5000, seed=0):
    """E: (N, d) array of unit-normalized chunk embeddings."""
    rng = np.random.default_rng(seed)
    i = rng.integers(0, len(E), n)
    j = rng.integers(0, len(E), n)
    keep = i != j
    sims = np.einsum("nd,nd->n", E[i[keep]], E[j[keep]])
    return np.percentile(sims, [5, 50, 95, 99])

Random pairs from the same corpus will slightly overestimate "unrelated" (since chunks share topic and terminology), but that's fine — you want a conservative floor.

Step 2: Labeled sample set. Write 40–50 questions your documents can answer, and 40–50 that they can't. Record the Top-1 score for each, then pick the threshold that achieves your required precision:

def pick_threshold(pos_scores, neg_scores, min_precision=0.95):
    candidates = sorted(set(pos_scores) | set(neg_scores))
    for t in candidates:
        tp = sum(s >= t for s in pos_scores)
        fp = sum(s >= t for s in neg_scores)
        if tp and tp / (tp + fp) >= min_precision:
            return t, tp / len(pos_scores)  # threshold, recall
    return None, 0.0

If no threshold achieves the target precision at reasonable recall, the problem isn't the threshold — your embeddings simply can't distinguish these questions, and you need a reranker or better chunking.

Store the model name and version alongside the threshold in your config. When someone upgrades the model, a stale threshold should fail loudly, not silently carry on.

Does Mean-Centering Fix Anisotropy?

For models with a strong shared direction, it helps significantly. Subtracting the corpus mean vector and re-normalizing opens up the cone. This is the idea behind the "all-but-the-top" post-processing approach (Mu and Viswanath, 2018), which also removes a few top principal components.

mu = E.mean(axis=0)

def center(x):
    x = x - mu
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

E_c = center(E)
q_c = center(q)  # always use the same mu for queries

The pitfalls are operational, not mathematical:

• Queries must be centered with the exact same mu as the corpus.
• mu drifts as the corpus grows; it needs to be recomputed periodically, which means re-embedding or re-centering the index.
• Your vector index (pgvector, FAISS, etc.) must store the centered vectors — this is an index rebuild, not a config toggle.

Centering makes scores more readable, but it doesn't make them calibrated. You still need to run Step 2.

"All-but-the-top" post-processing (Mu & Viswanath, 2018) is a lightweight fix specifically targeting anisotropy. The full version has two steps: first subtract the corpus mean vector (eliminating the common offset direction), then identify the top-D principal component directions of the covariance matrix and project them out of each vector (D is typically 1–5). Just doing the first step — mean subtraction — already captures most of the benefit.

The reason it works: the root cause of anisotropy is that a few dominant directions control the vector space. Subtracting the mean and removing principal components forces vectors away from that narrow cone, letting the remaining directions carry semantic differences more evenly. On word vectors and some sentence embeddings, this post-processing can boost word analogy task accuracy by several percentage points and significantly lower the cosine similarity baseline for random pairs. But as noted above, it addresses "score readability" — it doesn't automatically give you a calibrated threshold. The two need to be used together.

What to Use as a Gate Instead of Raw Scores

Use signals that survive a model swap. Three that the author found reliable:

1. Score vs. noise floor. Only accept a Top-1 hit when it exceeds the 99th percentile of random-pair similarity. When the model changes, recompute the floor, and the gate moves with it.
2. Top-1 margin. When a query genuinely matches something, the Top-1 hit typically stands apart from the next few results; when nothing matches, scores are flat. top1 - median(top2..top10) is a cheap, scale-invariant relative signal.
3. Cross-encoder reranker for the final relevance decision. It reads query and chunk together and directly judges relevance, rather than comparing two independent vectors. Its scores also need calibration (many output raw logits), but it draws a much cleaner line between relevant and irrelevant.

Cross-encoder rerankers differ architecturally from dual-encoder embedding models in a fundamental way. Embedding models are "bi-encoders": query and document are encoded independently into vectors, similarity is computed via dot product or cosine, and documents can be pre-indexed for fast retrieval. A cross-encoder concatenates query and document and feeds them through the model together, using full attention interaction to directly output a relevance score. It can't be pre-computed, has higher latency, but is far more accurate than vector similarity.

In RAG systems, the common pattern is a two-stage "coarse retrieval + fine reranking" pipeline: first use embedding vectors to retrieve the top 20–100 candidates (recall stage), then use a cross-encoder to rescore those candidates and take the top 3 (reranking stage), and finally pass only the reranked results to the LLM for generation. This preserves the speed of vector retrieval while gaining the semantic precision of cross-encoders. Popular open-source cross-encoders include the cross-encoder/ms-marco-MiniLM family; hosted APIs like Cohere Rerank are also an option. Their relevance scores still need threshold calibration for your specific task, but because the model directly models "relevant vs. not relevant," the scores are far more stable in meaning than cosine similarity.

Checklist for Every Embedding Model Swap

• Re-embed a sample, plot the random-pair distribution, record p50 and p99.
• Re-score your labeled set, re-select the threshold, and commit it alongside the model name.
• Check vector norms. If they're not close to 1.0, normalize — or consciously switch to dot product.
• Diff Top-1 results between old and new on the labeled set. Changes in ranking are the real migration risk.
• Alert when the "I don't know" rate spikes. That metric could have caught the failure in under an hour instead of a full day.

So What Does Cosine Similarity 0.8 Actually Mean?

In isolation: nothing. A cosine threshold like 0.8 only has meaning relative to a specific embedding model on a specific corpus — because anisotropy and contrastive training give every model its own score scale and noise floor. Unrelated texts might score 0.7 on one model and 0.15 on another.

Trust the ranking your vector retrieval gives you. Treat that number as uncalibrated until you've measured random-pair similarity in your own index and selected a threshold from labeled examples. Then, every time you swap models, do both of those things again.