EmbeddingGemma 2 Hands-On: A Multimodal Retrieval Model That Runs on Your Phone

Google's EmbeddingGemma 2 maps text, code, images, video, and audio into a single shared vector space — and it's small enough to run on a smartphone. For developers building multimodal RAG (Retrieval-Augmented Generation) pipelines, this open-weight model deserves serious attention. Beyond retrieval, it supports cross-modal classification and clustering using the same model.

Why Multimodal Retrieval Matters

Traditional RAG is almost entirely text-based: split documents into chunks, embed each chunk, then find the closest vectors at query time. But real-world data goes far beyond text — photos on your phone, screen recordings, voice memos, meeting recordings, slide decks, scanned documents. Pure text retrieval is completely blind to all of this.

Consider this: you want to find the moment in a video where "a dog catches a frisbee," but the video has no text description whatsoever. A text-based retrieval system simply can't surface that clip. This is especially critical for AI agents, whose memory is only as good as what they can retrieve. Multimodal retrieval is the foundational layer for any serious search application.

How It Was Solved Before

One approach is "convert everything to text": run an image captioning model on each photo, run speech-to-text on each audio clip, then embed the resulting text. The problem is you're chaining three models together — and more critically, you can only retrieve what the description or transcript actually says. If the captioning model writes "a dog in the park" without mentioning the frisbee, your search will miss it.

Google's Gemini Embedding paper quantified this gap: directly embedding audio scored 73.99 on a sound retrieval benchmark, while transcribing first and then embedding only scored 70.40.

OpenAI's CLIP (2021) used two separate models for images and text, pulling photos and their descriptions close together in vector space — great for image-text matching, but limited to one modality pair and trained on short captions. Meta's ImageBind (2023) extended this idea to six modalities. ColPali took a different angle, embedding page screenshots directly to bypass OCR — but storing many vectors per page causes indexes to balloon quickly. The latest approach uses a single backbone network to process all modalities. That's exactly how Gemini Embedding works, but it's an API-only model. EmbeddingGemma 2 brings the same design under one billion parameters with open weights.

CLIP (Contrastive Language-Image Pre-training), introduced by OpenAI in 2021, pioneered zero-shot image-text matching using contrastive learning: jointly training an image encoder and a text encoder so that "a photo of a cat" and "a cat" land close together in vector space, while unrelated pairs stay far apart. However, it's architecturally limited to two modalities and primarily trained on short English descriptions. ImageBind improved on this by incorporating audio, depth maps, thermal imaging, and IMU sensor signals — aligning all modalities to images as a "hub" — but still relies on images as an intermediary rather than a true end-to-end multimodal backbone. ColPali takes a different approach entirely: feeding document pages as screenshots directly into a vision-language model and generating multiple local vectors per image region (similar to token-level representations), bypassing OCR and layout parsing at the cost of storing hundreds of vectors per page, making it impractical for large-scale document libraries.

How the Model Works Internally

An embedding is a sequence of numbers that captures the meaning of an input — similar inputs produce similar numbers. Since text models can't directly process pixels or audio waveforms, EmbeddingGemma 2 uses two small front-end encoders: a 170M-parameter vision encoder that converts images or video frames into tokens, and a 300M-parameter audio encoder that does the same for audio. These tokens are fed into the same sequence as text tokens.

From the model's perspective, a photo is just a few hundred extra tokens in the input. By default, one image takes up 280 tokens, one video frame takes 140, and one second of audio takes about 25 tokens. The context window is 8,192 tokens, so a single input can hold roughly 29 images, 58 video frames, or about 5.5 minutes of audio. You can even embed a product description, photo, and short video clip together into a single vector.

All of this passes through a shared backbone — a Gemma-based 270M-parameter text model. The model then applies mean pooling across all token outputs to produce a single vector. Google hasn't released a technical report for this model, but states it uses the same techniques as Gemini Embedding: contrastive training that pulls paired content together (a photo and its caption, a clip and its transcript) while pushing unrelated content apart, repeated across massive mixed-modality datasets until everything lands in the same shared space.

Contrastive Training is the dominant training paradigm for multimodal embedding models today. The setup involves constructing "positive pairs" (e.g., a photo and its caption) and "negative pairs" (unrelated photo-text combinations from the same batch), with the optimization objective of maximizing cosine similarity for positive pairs and minimizing it for negative pairs, typically using InfoNCE loss. The key challenge lies in selecting "hard negatives": if negatives are too easy to distinguish, the model learns no fine-grained semantics; but if genuinely similar but unpaired samples are treated as negatives, they produce noisy gradients. Large-scale training also requires very large batch sizes or dedicated negative mining strategies. Google hasn't published a technical report for EmbeddingGemma 2, but the referenced Gemini Embedding paper indicates the model series uses carefully curated multilingual, multimodal paired data with a curriculum learning schedule (easy examples first). Mean pooling — averaging the hidden states of all tokens at the end of a sequence — is more stable for longer inputs compared to using only the [CLS] token representation.

Two Key Deployment Design Decisions

The first is Matryoshka Representation Learning. The model is trained so that vectors truncated to 256, 128, or even fewer dimensions still function as valid embeddings. This means you can cut vectors down for smaller indexes. The trade-off is real: according to Google's own data, text performance barely changes at 128 dimensions, but multimodal scores drop from 59 down to about 46.

The second is modular loading. The vision and audio encoders are optional. Text and code only: 270M parameters. Add vision: 470M. Add audio: 570M. Full model: 740M. They all share the same vector space, so you can index photos with the full model and then search on a phone with the lightweight text-only model. With quantization, Google reports the text-only version uses about 191MB of RAM on a Pixel phone; the full model uses about 567MB.

Matryoshka Representation Learning (MRL) is named after Russian nesting dolls and was proposed by Kusupati et al. in 2022. The core idea is to simultaneously apply supervised loss at multiple prefix lengths of the embedding vector (e.g., 2048, 1024, 512, 256, 128 dimensions) during training, forcing the model to concentrate the most important semantic information into the leading dimensions. This means the full-length output vector can be directly truncated for use without retraining a separate smaller model. In practice, this enables a two-stage "retrieve-then-rerank" architecture: use short vectors for fast, storage-efficient first-stage retrieval, then use full vectors for precise reranking of candidates — balancing latency and accuracy. The steeper accuracy drop for multimodal tasks at smaller dimensions occurs because cross-modal alignment requires more information than pure text semantics, leaving less room for compression.

Hands-On: Running It in Colab

The author ran the full pipeline on a free T4 GPU in Colab in about 15 minutes. Two gotchas worth noting: don't run this model in float16 — the model card explicitly warns of activation overflow, and T4 GPUs don't natively support float16 anyway. You'll get NaN values or corrupted embeddings. Use float32, or bfloat16 on newer GPUs.

Test data included 1,000 Flickr photos with human-written captions, 50 everyday sound clips from ESC-50, the first five minutes of the open-source film Big Buck Bunny, and the Attention Is All You Need paper in PDF.

Image retrieval was the standout result. Each photo was embedded once (about 8 minutes total), and each subsequent search required only one text embedding plus one comparison. Across 5,000 queries with ground-truth answers, the correct photo ranked first about 86% of the time, and appeared in the top 5 about 97% of the time — impressive for a model of this size. Thanks to Gemma's multilingual capabilities, the text side supports over 100 languages; queries in German and Spanish returned correct results. You can even query with an image: submitting a surfing photo returned other surfing and wakeboarding photos — matching on "person balancing on a board over water" rather than just the color blue.

Voice search demonstrated true cross-modal capability: generating a voice clip saying "a dog jumping into a lake" and feeding it directly to the model returned the exact same four photos as typing the same text query — no speech-to-text step needed.

Weaknesses: Ambient Sound and Small Dimensions

The main weakness is ambient sound retrieval. Across 50 everyday sound clips, the correct clip ranked first only 24% of the time (random chance would be 2%). Laughter found laughter, flushing found toilets — but "dog barking" returned crows, and "rain" returned wind. The audio encoder is strongest on speech, which aligns with the voice search demo. If you need to retrieve sound effects or ambient audio, test carefully on your own clips first.

Video retrieval worked quite well: splitting the first five minutes into 5-second clips, embedding each one (defaulting to one frame per second), and running text queries returned correct timestamps. "A bird spreading its wings on a branch" landed at 15 seconds; "purple butterfly" landed at 3 minutes 20 seconds — all with no subtitles or captions needed. Document retrieval was similarly effective: converting each page of the paper to an image and embedding it directly, a query for "Transformer architecture diagram" returned page 3 with the famous figure, and "multi-head attention" returned page 4.

On the practical impact of Matryoshka truncation: at full dimensions, the correct photo ranked first 86% of the time. At 256 dimensions that dropped slightly to 84% with 3x smaller storage. At 128 dimensions it fell to 75% — past a certain threshold, the accuracy drop becomes noticeable and needs careful evaluation for your use case.

Conclusion

EmbeddingGemma 2 delivers strong photo, video, and document retrieval using a single open-source model under one billion parameters. It supports indexing with the full model and searching with the lightweight text-only version on a mobile device. For developers building search or RAG applications, this is the first open-source multimodal embedding model worth trying first. Ambient sound retrieval isn't mature yet, but the model can be fine-tuned for specific use cases. Its ability to run on a phone while covering multiple modalities opens up significant possibilities for local private photo search, agent memory systems, and much more.