Matryoshka Representation Learning
An embedding model training technique that makes the first several dimensions of a single model (such as 128, 256, or 512) each constitute a usable embedding on their own, allowing lower dimensions to be traded for smaller storage and faster retrieval without retraining.
Timeline (last 90 days)
EmbeddingGemma 2 uses Matryoshka vector representations, where the first 128, 256 or 512 of the 768 dimensions can be used independently as embeddings.
The cost of Matryoshka representation learning is that the lower the truncated dimension, the more pronounced the drop in retrieval quality, requiring a trade-off between recall and storage cost.
The nested model technique follows the same line of thinking as the academic Matryoshka representation learning approach, training model subsets to be independently usable rather than pruning after the fact.
All Facts (3)
EmbeddingGemma 2 uses Matryoshka vector representations, where the first 128, 256 or 512 of the 768 dimensions can be used independently as embeddings.
50%UnverifiedThe cost of Matryoshka representation learning is that the lower the truncated dimension, the more pronounced the drop in retrieval quality, requiring a trade-off between recall and storage cost.
50%UnverifiedThe nested model technique follows the same line of thinking as the academic Matryoshka representation learning approach, training model subsets to be independently usable rather than pruning after the fact.
50%