Embeddings from First Principles: How a Kitten Finds a Cat, and Even Crosses Languages to Find Gato

Computers cannot compare meaning directly — only numbers. This article breaks down embedding models from first principles: how a piece of text becomes a vector, how cosine similarity measures semantic distance, and why using the wrong model makes multilingual RAG fail silently.

Why keyword search breaks down

Keyword search inevitably fails in two scenarios: synonyms and translation. Search for "kitten" and the document says "cat"; search for "cat" and the document says the Spanish word "gato". There is no overlap at the character level, so the inverted index finds nothing.

To solve this, you need a way to turn meaning into numbers. That is exactly what embeddings do, and it is the first step of the "R" (retrieval) in RAG.

What an embedding is: understanding vectors through a color analogy

An embedding model converts any text into a fixed-length list of numbers — a vector. Every color on your screen is just 3 numbers (R, G, B):

Orange       → [255, 165,   0]
Dark orange  → [255, 140,   0]   ← numbers are close to orange
Blue         → [  0,   0, 255]   ← numbers are far from both

Orange and dark orange look similar, and their numbers are close. Blue looks different, and its numbers are far away. Three numbers can describe 16 million colors, but 3 numbers do not equal 3 colors.

Embeddings do the same thing, just with more numbers. Different models use different dimensions:

Model	Numbers per text
paraphrase-multilingual-MiniLM-L12-v2	384 (used in this article)
BERT-base series	768
OpenAI text-embedding-3-small	1536
OpenAI text-embedding-3-large	3072
Google gemini-embedding-001	3072

Higher dimensions capture finer detail, but cost more storage and slow down retrieval. Here is what a 384-dimensional model produces:

"The cat sat on the mat."         → [ 0.43, -0.38, -0.23, ... ]
"El gato duerme en la alfombra."  → [ ... similar numbers ... ]   ← same meaning, close distance
"Stock markets fell sharply."     → [ ... completely different numbers ... ] ← unrelated meaning, far distance

Two key facts are easy to overlook: 384 is the model's output dimension, determined by its design, independent of the text's word count or character count. And a single number means nothing on its own — meaning only emerges when vectors are compared with each other.

Cosine similarity: measuring semantic distance by angle

Cosine similarity measures the angle between two vectors: a result of 1 means the same direction (same meaning), and 0 means unrelated.

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

It ignores vector length, so a short question can still match a long document.

Semantic retrieval in twenty lines of code

from sentence_transformers import SentenceTransformer
import numpy as np

docs = [
    "The cat sat on the mat.",
    "Dogs are loyal pets.",
    "Stock markets fell sharply today.",
    "El gato duerme en la alfombra.",   # Spanish: the cat sleeps on the rug
    "The central bank raised interest rates.",
]
query = "Where is the kitten?"

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
doc_vecs = model.encode(docs)
q_vec = model.encode(query)

scores = [cosine(q_vec, d) for d in doc_vecs]
for score, doc in sorted(zip(scores, docs), reverse=True):
    print(f"{score:.3f}  {doc}")

Output:

0.560  The cat sat on the mat.
0.480  El gato duerme en la alfombra.     ← Spanish sentence ranks second
0.138  Dogs are loyal pets.
0.049  Stock markets fell sharply today.
0.046  The central bank raised interest rates.

"kitten" found "cat", with no shared words between the two sentences. The Spanish sentence took second place. The model never compares "gato" and "cat" letter by letter — it was trained on millions of translation pairs with the goal of placing each pair at the same position in vector space. Where a sentence lands depends on meaning, not language.

In the code above, paraphrase-multilingual-MiniLM-L12-v2 is a multilingual sentence embedding model provided by the Sentence Transformers library. It compresses the output of a 12-layer Transformer encoder into a 384-dimensional vector through mean pooling, and supports about 50 languages. Sentence Transformers is a framework for contrastive learning fine-tuning on top of pretrained models such as BERT and XLM-R, with the training goal of bringing semantically similar sentence vectors closer together. The model's weights are downloaded automatically by encode() on first use. What truly gives it cross-lingual ability is the multilingual XLM-R backbone combined with training data of millions of translation pairs (such as English–Spanish translations): synonymous cross-lingual sentence pairs are pulled into the same region of vector space during training, so the model can compare semantics without relying on shared words.

Multilingual capability gone wrong: the illusion of spelling overlap

The same test with all-MiniLM-L6-v2, trained only on English, flips the results:

Model          Query                        cat (EN)  gato (ES)
Multilingual   "Where is the kitten?"       0.560     0.480
Multilingual   "¿Dónde está el gatito?"     0.569     0.467
English-only   "Where is the kitten?"       0.418     0.096
English-only   "¿Dónde está el gatito?"    -0.069     0.616  ← the trap

The last line is the trap: the English-only model gives the Spanish sentence the highest score, but not because it understood it. Look at its tokenizer output:

query: ['¿', 'don', '##de', 'est', '##a', 'el', 'ga', '##ti', '##to', '?']
doc:   ['el', 'ga', '##to', 'due', '##rm', '##e', 'en', 'la', 'alf', '##om', '##bra', '.']

They share the fragments el, ga, and ##to, so this is spelling overlap, not semantic understanding. At the same time, it ranks the English cat sentence last.

The conclusion is straightforward: when the query and document share spelling, the model looks multilingual. Only a cross-lingual test (ask in language A, answer in language B) can prove it truly is. This is also the classic way RAG fails silently after choosing the wrong model.

Here, ## is the subword prefix marker of the WordPiece tokenizer. BERT-family models first split words into subwords that exist in the vocabulary: if a full word is not in the vocabulary, it falls back to shorter fragments and marks them with ## to indicate they are the latter part of a word. For example, gatito is split into ga, ##ti, ##to, and gato also contains ##to, so the two share tokens between the query and the document. This mechanism lets the model handle out-of-vocabulary words, but it also has a side effect: languages with similar orthography (such as Latin-alphabet English and Spanish vocabulary) produce false similarity due to fragment overlap, while the model actually outputs an encoding of spelling fragments rather than full semantics.

Two easy pitfalls

Do not compare vectors across models. Every model has its own "semantic map". Comparing a vector from model A with a vector from model B is like comparing GPS coordinates with RGB values. Once you switch models, all documents must be re-embedded.

Watch the maximum length. This multilingual model reads at most 128 tokens; anything beyond that is silently truncated. This is exactly why RAG splits documents into chunks. You can find the maximum sequence length in the model's downloaded config file:

cd ~/.cache/huggingface/hub/models--sentence-transformers--paraphrase-multilingual-MiniLM-L12-v2/snapshots/*/ 
cat sentence_bert_config.json

# Output:
# {
#   "max_seq_length": 128,
#   "do_lower_case": false
# }

The tokenizer is the model's "eyes". If it splits a language into semantically meaningless fragments, the model can only match spelling, not meaning.

Additional background

max_seq_length is the maximum number of tokens the model allows during encoding. Here, 128 tokens corresponds to roughly 80 to 100 English words, or fewer for non-Latin-alphabet languages. Sentence Transformers truncates overly long text to this length by default during encoding, without warning, so the latter half of longer document chunks is silently discarded, and retrieval behaves as if that content never existed. The common 256- or 512-token chunking strategies in RAG exist precisely to match this upper limit across different embedding models. Queries are subject to the same limit: overly long queries get truncated, causing key information loss. When choosing a model, besides dimensions, you should also confirm whether max_seq_length covers your typical document chunk length.