Google Open-Sources EmbeddingGemma2: Text and Images in the Same Vector Space

A Photo Search Experiment With No Labels

Bilibili creator Steven opens with a compelling experiment: 300 animal photos from the public dataset Animals with Attributes 2, with no text labels, filenames, or category information. Type "animals that can fly" into the search box, and the top 9 results are all bats — some in flight, some hanging upside down from trees. Search for "black and white animals" and you get zebras, pandas, and a leopard with dark spots.

The key point: these photos carry no embedded color metadata. The model is directly "reading" the visual content of each image to make its judgments. This is the core capability of Google's latest open-source multimodal Embedding model — EmbeddingGemma2.

Putting Text and Images Into the Same Space

The basic idea behind Embedding models is this: convert a sentence into a series of numbers, and two sentences with similar meanings will have vectors pointing in similar directions. EmbeddingGemma2 extends this concept — it doesn't just read text. It can also process images, audio, and video, converting all of these different modalities into 768 numbers each.

Text and images have their own separate encoders, but both output 768-dimensional vectors in the same shared space. This means a sentence can be directly compared against an image. Steven demonstrated this with 10 photos each of five animals (tiger, panda, giraffe, dolphin, elephant): when compressed to 2D, each animal species clusters together; adding three text sentences, the three closest photos for each sentence point to the correct animal.

He does note an important caveat: the 2D map is only an approximation. Real distance calculations must use the full 768-dimensional vectors.

Embedding models convert content into high-dimensional vectors, with the core goal of placing semantically similar content close together in mathematical space. EmbeddingGemma2's multimodal architecture is a variant of the Dual Encoder model: text is processed by a language encoder, images by a visual encoder (typically based on ViT, Vision Transformer), and both channels encode independently before outputting into a shared vector space. This alignment capability comes from contrastive learning during training — paired text and image vectors are pushed close together, while unpaired ones are pushed apart. OpenAI's CLIP model was an early representative of this approach; EmbeddingGemma2 is the latest iteration, built on the Gemma 2 language model architecture and trained with multimodal contrastive learning. The 768-dimensional output is an engineering tradeoff: higher dimensions offer greater expressiveness, but storage and compute costs scale accordingly. 768 dimensions represents a common industry sweet spot between precision and efficiency.

Real-World Search: Why Do the Scores Look So Strange?

Steven deployed EmbeddingGemma2 on his DGX Spark, which exposes an OpenAI-compatible API. The workflow is straightforward: precompute vectors for all 300 photos and store them in a JSON file (each entry contains only the photo path and a 768-dimensional vector, no text descriptions), then at search time simply embed the query text into a query vector and compute the dot product against each photo's vector.

Since the vectors are length-normalized to 1, the dot product equals cosine similarity. But running "animals that can fly" revealed a counterintuitive result: if you set a score threshold above 0.8, no photos appear at all.

Plotting all 300 photo scores on a single axis, the highest-scoring bat photo only reached 0.756, with all photos squeezed between 0.6 and 0.76. By contrast, the similarity between two bat photos was as high as 0.951 — and even a bat versus a dolphin scored 0.781, higher than the "correct text matched to correct photo" score.

Why Cross-Modal Scores Are Systematically Low

The reason: even though text and images share the same vector space, they tend to cluster in different regions of that space. So text-to-image scores are consistently lower overall. This yields an important practical insight — don't use a fixed score threshold to filter cross-modal search results; just sort by rank. Removing the threshold and sorting by score descending to take the top 10 still returns bats as the first 9 results. Searching for "animals swimming in water" returns beavers, hippos, dolphins, and a grizzly bear soaking in water — perfectly ranked.

This phenomenon — where text and image vectors still cluster in separate regions of a shared space — is known as the Modality Gap in multimodal retrieval. Even after contrastive learning alignment, vectors from different modalities tend to occupy distinct cone-shaped regions of high-dimensional space rather than mixing uniformly. This was systematically analyzed in the 2022 research paper Mind the Gap, with causes linked to encoder architecture differences, training data distribution, and the geometric properties of high-dimensional spaces. Beyond "sort by rank," practical mitigations include per-query normalization of within-modality scores, or training a lightweight Reranker specifically for cross-modal retrieval to calibrate score distributions.

Truncating Vectors to Save 3× Storage

One notable feature of EmbeddingGemma2 is support for vector truncation. With 768 numbers per photo and 4 bytes per number, 1 million photos requires about 3 GB. Because the model is trained to encode the most important information in the earliest dimensions of the vector (the core idea behind Matryoshka Representation Learning), you can keep only the first 256 or even 128 numbers. Google reports that searching with 256 dimensions retains approximately 95% of quality.

Implementation requires only a shorten function using slice to take the first n numbers. One critical note: after truncation, the vector length is no longer 1. The first 256 dimensions have a length of approximately 0.709, so you must divide each number by the vector length to re-normalize. Truncating both query and photo vectors to 256 dimensions reduces data size by 3× (1 million photos drops from ~3 GB to ~1 GB), and the top 9 results are identical to the 768-dimensional version.

Truncating further to 128 dimensions lets squirrels, foxes, and hippos creep in, while bats drop to only 7 of the top results. Steven's advice here is pragmatic: how short you can cut depends on your own data — there's no universal answer.

Matryoshka Representation Learning (MRL) takes its name from Russian nesting dolls. The core idea is to simultaneously optimize performance across multiple dimensional subsets during training, forcing the model to encode the most discriminative information into the earliest dimensions first. Traditional Embedding models truncated to the first 256 dimensions typically suffer significant quality degradation, because important information is spread uniformly across all dimensions. MRL addresses this with a multi-scale loss function, making the first 128, 256, or 512 dimensions each independently usable as complete vectors. This is extremely practical in production: you can use short vectors for coarse filtering, then re-rank candidates using the full 768-dimensional vectors, flexibly trading off precision against speed. Google has applied this technique in models like text-embedding-004, and EmbeddingGemma2 extends it to multimodal scenarios.

Summary

EmbeddingGemma2 converts text and images into 768-dimensional vectors in a shared space, enabling a natural language sentence to directly search through photos. Three key engineering takeaways: precompute photo vectors in advance and at search time simply embed the query and sort by dot product; cross-modal scores are systematically low, so sort by rank rather than applying a fixed threshold; vectors can be truncated to save space, but must be re-normalized after truncation. For applications that need to build image-text retrieval, image-to-image search, or natural language image search, this is an open-source tool worth experimenting with.