MAEB
The Massive Audio Embedding Benchmark, a large-scale benchmark used to evaluate model performance on audio embedding and retrieval tasks.
Timeline (last 90 days)
By projecting the features of an intermediate Conformer layer into the null space of the model's final projection matrix, without any training, EmbeddingGemma-2's ranking on the MAEB audio benchmark can jump from 26th to 1st.
Existing LLM-driven Omni embedding models score 95-99% on multilingual text-to-audio retrieval on the MAEB leaderboard, but only 1%-6% on unsupervised audio clustering and about 20% on speech emotion recognition (CREMA-D).
Using PCA to compress layers 2 and 6 to 256 dimensions (fitted on 1400 clips from 4 external audio datasets, without using MAEB/MSEB data), injecting into the null space and normalizing, text embeddings remain bit-for-bit unchanged, and text-to-audio retrieval changes by within 0.05 points.
The overall MAEB macro average rises from 49.3% to 54.4% (+5.1 points), topping the pure-audio track from 26th place and rising from 16th to 1st on the full MAEB, ahead of 8B-9B multimodal models.
EmbeddingGemma 2 achieves leading results among multimodal embedding models of the same size on benchmarks such as MTEB Code and MAEB, setting a new standard for quality per parameter among models with fewer than one billion parameters.
All Facts (5)
By projecting the features of an intermediate Conformer layer into the null space of the model's final projection matrix, without any training, EmbeddingGemma-2's ranking on the MAEB audio benchmark can jump from 26th to 1st.
50%UnverifiedExisting LLM-driven Omni embedding models score 95-99% on multilingual text-to-audio retrieval on the MAEB leaderboard, but only 1%-6% on unsupervised audio clustering and about 20% on speech emotion recognition (CREMA-D).
50%UnverifiedUsing PCA to compress layers 2 and 6 to 256 dimensions (fitted on 1400 clips from 4 external audio datasets, without using MAEB/MSEB data), injecting into the null space and normalizing, text embeddings remain bit-for-bit unchanged, and text-to-audio retrieval changes by within 0.05 points.
50%UnverifiedThe overall MAEB macro average rises from 49.3% to 54.4% (+5.1 points), topping the pure-audio track from 26th place and rising from 16th to 1st on the full MAEB, ahead of 8B-9B multimodal models.
50%UnverifiedEmbeddingGemma 2 achieves leading results among multimodal embedding models of the same size on benchmarks such as MTEB Code and MAEB, setting a new standard for quality per parameter among models with fewer than one billion parameters.
50%