Fine-Tuning Embedding Models Is Easier Than You Think: A Hands-On Guide to EmbeddingGemma 2

Out-of-the-box embedding models only understand the generic data they were trained on — they know nothing about your internal documents, product-specific terminology, or the particular sounds in your application. EmbeddingGemma 2, as a strong open-source multimodal retrieval model, offers something valuable: you can fine-tune it on your own data. One YouTube tech creator proved it in practice — a few minutes of fine-tuning can meaningfully improve a model's performance on your specific data, with minimal damage to its existing capabilities.

The Fundamental Difference Between Fine-Tuning Embedding Models and LLMs

To understand embedding model fine-tuning, you first need to grasp how it fundamentally differs from fine-tuning large language models. When you fine-tune an LLM, you provide an input alongside a target text you want the model to reproduce word for word — the training data contains the "right answer."

Embedding models don't output any text. They simply convert inputs into a sequence of numbers (vectors), and the only meaning those numbers carry lies in their relative relationships to one another — what people call the "embedding space." There is no "correct vector" you can write down as a target. The only thing you can tell the model is which things should be close to each other.

This changes the shape of your training data. Imagine you're building a customer support search box for a smart home company, designed to answer questions from a help center. Each training sample is a pair: a customer question and the document passage that answers it. You don't need scores, you don't need labeled wrong answers — you only need question-and-passage pairs that "belong together."

What is an embedding space? An embedding model maps any input — text, images, audio — to a high-dimensional vector, typically an array of floating-point numbers with hundreds to thousands of dimensions. The vector itself has no human-readable meaning, but its position in space encodes semantics: content with similar meaning gets mapped to nearby regions, while unrelated content gets pushed far apart. The standard way to measure whether two vectors are "close" is cosine similarity — the cosine of the angle between the two vectors, where 1 means identical, 0 means orthogonal, and -1 means opposite. Fine-tuning an embedding model is essentially about adjusting how your specific data clusters together, without destroying the overall structure of the space.

Free Negatives: Multiple Negatives Ranking Loss

Simply pulling a question toward its corresponding passage isn't enough — the model could cheat by mapping everything to a single point and still score well. What matters is how the model learns what to push away and what to pull close.

During training, the model processes a batch of pairs at once — say, 32 pairs per batch. For each question, its corresponding passage is the correct answer, and all other passages in the batch are treated as wrong answers. This means you never need to manually collect negative samples — they come for free. The loss function pulls each question toward its own passage while pushing it away from all others, giving you "wrong answers" without labeling a single negative example.

The most intuitive analogy is a multiple-choice question, where the other pairs in the batch serve as distractors. In sentence-transformers, this is called multiple negatives ranking loss, and it's the same idea CLIP uses to align images and text in a shared space.

Don't Fall Into the Duplicate Sample Trap

There's a pitfall here: if two customers ask about the same reset process and both pairs end up in the same batch, the loss function will treat the second identical passage as a wrong answer for the first question — essentially teaching the model that the correct answer is wrong. The fix is a batch sampler that never places the same text in the same batch, which sentence-transformers calls a no-duplicate sampler.

Prompt Templates Must Be Consistent Between Training and Inference

The second key point is prompts. EmbeddingGemma 2 requires a short task prompt before search queries, and documents use a "title + body" format. Whatever format you use during training must be used exactly in production — just like prompt templates for LLMs, the training and inference templates must be identical.

The math behind Multiple Negatives Ranking Loss (MNRL) draws from the contrastive learning framework. Given a batch of N pairs, each query q_i and its corresponding document d_i form a positive example, while the remaining N-1 documents d_j (j≠i) in the batch serve as "in-batch negatives." The loss function requires the model to score the positive example higher than all negatives, typically implemented with cross-entropy: similarity scores are passed through softmax, and the model is trained to maximize the probability of the positive example. This is structurally identical to OpenAI CLIP's training objective — CLIP symmetrically computes the loss in both image→text and text→image directions. The larger the batch, the more negatives available, and the stronger the supervisory signal — which is why large-batch training is especially impactful for embedding models.

LoRA Makes Training Fast and Efficient

What actually gets trained? This is the most interesting part. Rather than updating all weights, the author used LoRA — inserting small trainable matrices alongside existing layers while keeping the original weights frozen. LoRA adapters train extremely fast without requiring the entire network to be retrained. In the author's experiments, trainable parameters were less than 2% of total weights, meaning you only need to save a small adapter file rather than a full copy of the model.

There's a noteworthy caveat: EmbeddingGemma 2's text, image, video, and audio modalities all share the same backbone. So if you train the backbone on audio, you'll also affect performance on other modalities — a trade-off we'll quantify with experiments later.

LoRA (Low-Rank Adaptation) is built on the insight that after large-scale pretraining, the actual changes in weights during downstream fine-tuning tend to be low-rank. Rather than directly updating the original weight matrix W (which might be 4096×4096), you insert two small matrices A (4096×r) and B (r×4096), where r is much smaller than 4096 (e.g., r=8 or r=16). During the forward pass, the output becomes Wx + BAx, with W frozen and only A and B participating in gradient updates. Since r is small, trainable parameters can be compressed to 0.1%–2% of the original, dramatically reducing memory usage and training time. After training, the LoRA adapter can be saved separately from the base model or merged into the original weights — once merged, inference speed is identical to the original model with no additional overhead.

Measuring Results the Right Way

A decreasing training loss only means the model got better on samples it has already seen. What you actually care about is search quality on questions it has never seen — otherwise, you're just measuring overfitting.

The solution is to hold out a test set and split it by source. If passages from the same article appear in both the training and test sets, the model has effectively already seen the answers. You need to split by article, by video, by recording. This also gets at the core question: did the model learn a generalizable skill, or did it just memorize the labels you provided?

Experiment 1: Everyday Sound Classification from 25% to 66%

The author ran experiments on Colab's A100 GPU using Unsloth's EmbeddingGemma 2 notebook. The first example addressed a weakness from a previous video — recognizing everyday sounds. The dataset used was ESC-50, which contains short clips of 50 everyday sounds like dog barking, rain, and chainsaws.

Classification becomes a retrieval problem here: the model embeds sound clips, embeds label sentences, and after training, selects the closest match. Before training, the model only had roughly a one-in-four chance of ranking the correct sound first — about 25% accuracy — and the model's built-in classification prompts performed even worse.

Training pairs consisted of "label sentence + audio clip" — which is exactly where the duplicate sample trap bites hardest, since each label maps to many clips. The author added LoRA to both the audio encoder and the backbone, trained for about seven and a half minutes on the A100, and the top-1 hit rate on clips the model had never heard jumped from about 25% to about 66%, with sounds like "toothbrushing" being identified correctly almost every time.

Did It Learn to "Hear," or Just Memorize Labels?

The author ran a critical follow-up experiment: this time, 10 sounds were completely excluded from training, then tested exclusively on those 10. The result — trained sounds improved as expected, but the 10 unseen sounds showed zero improvement. The conclusion is clear: the model learned the labels you gave it, not a general auditory capability. This means that if you fine-tune for your own application, every label you care about must appear in the training data — generalization does not happen automatically.

The Cost of a Shared Backbone — and a Hidden Bug

What did fine-tuning cost? The author loaded the full model with and without the adapter and ran three checks: image search, voice search, and general text search. Image search and voice search were completely unaffected, but general text search dropped by about 4 points — the exact cost of modifying the shared backbone. If your application also relies on general text search, it's worth running this check after every fine-tuning run.

There's also a bug worth documenting: on the first check, voice search nearly dropped to zero, making it look like the model had completely forgotten audio. Stepping through the issue revealed that the adapter weights had loaded correctly — the problem was the processor saved alongside the adapter, which was trimming every audio clip to just a few milliseconds, making all sounds appear identical. The fix was a single line of code: reuse the base model's processor. So if you save and reload an audio adapter, always re-test after reloading.

Experiment 2: Turning YouTube Captions into a Searchable Knowledge Base

The second example is more practically applicable. The author took captions from 120 of his own videos and split them into short passages. Since no ready-made questions existed, he used Gemini 3 Flash to generate "viewer-style" questions for each passage, producing question-passage pairs in exactly the format described earlier. He then split by video, with test questions coming from videos the model had never trained on.

Before training, the correct passage ranked first about two-thirds of the time. After less than three minutes of training, that improved to roughly three-quarters. General text search dropped by about 4 points — the same cost as the audio experiment. The final result is the search box shown at the beginning of the video: enter a question, get back the exact moment in the video where that topic is discussed, with a clickable timestamp. To use this with your own data, you just need a file of question-passage pairs in the same format.

Using a language model to automatically generate training questions is commonly called "synthetic data generation" or "query generation" — a standard strategy for reducing annotation costs in embedding model fine-tuning. The logic is: if you have many documents but few real user queries, you can have a capable generative model role-play as a "potential user" and write natural language questions for each passage. The closer the generated questions are to real user phrasing, the better the fine-tuning outcome. In practice, watch out for diversity — avoid letting the model repeatedly generate structurally identical questions, which would leave the training data covering a very narrow range of language patterns. Also, the quality ceiling for synthetic questions is bounded by the generating model's comprehension, so for highly specialized domains, mixing in some real user queries for calibration is still advisable.

Three Pieces of Advice for Anyone Fine-Tuning EmbeddingGemma 2

The verdict is clear: a few minutes of fine-tuning does meaningfully improve the model on the author's sounds and videos. But it learns labels, not general skills, and it pays a small cost on general text search.

If you're planning to fine-tune EmbeddingGemma 2, the author recommends focusing on three things:

• Organize your data into pairs and ensure no duplicates appear in the same batch — use a no-duplicate sampler to avoid teaching the model that correct answers are wrong.
• Test on data it has never seen — split training and test sets by source to distinguish generalization from overfitting.
• Check what else changed after training — because the backbone is shared across all modalities, fine-tuning on one modality may affect others.

This is perhaps one of the most practical things you can do with an embedding model — at very low cost, achieve a meaningful performance boost on your own specialized data.