[KongchangAI]
· 2 min read· 1,362 words

One Small Model for Four Central Asian Languages: A Multilingual Embedding Fine-Tuning Practice with EmbeddingGemma

One Small Model for Four Central Asian Languages: A Multilingual Embedding Fine-Tuning Practice with EmbeddingGemma

A small embedding model fine-tuned on synthetic data supports four Central Asian languages with a big boost in top-1 retrieval accuracy.

This article presents Saidzoda-CA-EmbeddingGemma-300M: building on EmbeddingGemma-300M, continued training for one epoch with 279,904 synthetic question-passage pairs (new Kazakh, Uzbek, and Kyrgyz data, Tajik review data, and English/Russian replay data) yields a single model serving four Central Asian languages. On Belebele, recall@1 for the four languages rises from 51.7%-60.8% to 75.2%-77.7%, a gain of 14.4 to 26.0 percentage points. The unified model's recall@1 gap versus single-language models on Kazakh, Uzbek, and Tajik is at most 2.8 percentage points, but Tajik legal recall@1 drops 4.4 percentage points. Compared with 560M-level XLM-RoBERTa embedding models, a recall@1 gap of 9.4-10.2 percentage points remains, and English and Russian performance regress by 4.2 and 3.0 percentage points, respectively.

The Central Asian Language Predicament for Small Embedding Models

EmbeddingGemma-300M is an open-source small embedding model that performs well on mainstream languages like English. But if we use it directly for retrieval in Tajik, Kazakh, Uzbek, and Kyrgyz, the results leave something to be desired. According to the original release data, on the Belebele reading comprehension dataset, the un-fine-tuned EmbeddingGemma-300M ranks the correct passage first (recall@1) for questions in these four languages only 51.7% to 60.8% of the time. That means for more than 40% of questions, users have to scroll down to find the answer.

For low-resource languages like those in Central Asia, this is a very real pain point: general-purpose embedding models don't get enough exposure to learn semantic representations for these languages. Training a separate embedding model for each language is costly, hard to maintain, and prone to uneven results.

rss source: Saidzoda-CA-EmbeddingGemma-300M: One Small Embedder for Tajik, Kazakh, Uzbek and Kyrgyz

Embedding Model refers to a class of neural network models that map text to dense vectors, whose outputs can be used for downstream tasks such as semantic retrieval, clustering, and classification. In retrieval scenarios, the system typically first encodes the query and candidate passages into vectors using an embedding model, then ranks them by vector similarity (e.g., cosine similarity) to return the most relevant passages. EmbeddingGemma-300M is a small open-source embedding model released by Google with approximately 300 million parameters; it supports multiple languages but its training data is dominated by high-resource languages. recall@1 is a retrieval evaluation metric denoting the proportion of results where the correct passage is ranked first; recall@10 denotes the proportion where the correct passage appears in the top 10. Low-resource languages (such as Tajik, Kazakh, etc.) lack large-scale corpora, making it difficult for general-purpose embedding models to learn sufficient semantic representations—this is the starting point for the fine-tuning work in this article.

One Model, Four Languages: Saidzoda's Fine-Tuning Strategy

The core idea of this work is: building on an existing Tajik fine-tuned model, train for one more epoch, using synthetic multilingual data to "pull in" Kazakh, Uzbek, and Kyrgyz. The training set contains 279,904 question-passage pairs, and its composition is quite deliberate:

  • Questions in Kazakh, Uzbek, and Kyrgyz are generated by the author's own language model against source passages;
  • A portion of Tajik data serves as "review" to prevent catastrophic forgetting;
  • English and Russian data serve as "replay" to maintain the model's capabilities on general languages.

The resulting model is called Saidzoda-CA-EmbeddingGemma-300M. It is not four separate models but a single model serving four languages simultaneously—a fairly pragmatic design for a small embedding model.

Catastrophic Forgetting refers to the phenomenon where a neural network rapidly loses previously learned knowledge when learning a new task or new data. When continually fine-tuning an embedding model, training only on new language data can cause the model's representational ability for the original language (e.g., Tajik) to degrade significantly. The strategy in this article is to mix Tajik "review" data and English/Russian "replay" data into the training set, so that the model still sees samples from the old distribution when updating parameters, thereby mitigating forgetting. Replay is a classic method in continual learning that maintains balanced multilingual performance by retaining or replaying samples from old tasks. Additionally, the "synthetic data" mentioned refers to question-passage pairs automatically generated by a language model, used to compensate for the lack of annotated data in low-resource languages.

Performance Gains: recall@1 Rises Substantially Across Languages

On the Belebele dataset, the improvement from this unified model is significant. recall@1 for the four languages each rose by 14.4 to 26.0 percentage points, ultimately landing between 75.2% and 77.7%. In other words, the previous hit rate of just over half at rank one is now close to three-quarters.

rss source: Saidzoda-CA-EmbeddingGemma-300M: One Small Embedder for Tajik, Kazakh, Uzbek and Kyrgyz

Compared with "single-language models trained separately under the same settings," this unified model's recall@1 gap on Kazakh, Uzbek, and Tajik is no more than 2.8 percentage points, and the confidence intervals for recall@10 overlap. However, the author also candidly notes: the unified model used more training data (since it covers four languages), so this comparison is not entirely fair. But at least it shows that joint multilingual training does not significantly hold things back.

Belebele is a multilingual reading comprehension dataset covering 122 languages, where each question comes with four candidate passages and the task is to select the passage most relevant to the question. In embedding retrieval evaluation, the question and candidate passages are typically encoded separately, similarity is computed and ranked, and recall@1 is used to measure whether the correct passage is ranked first. Belebele's questions are designed to resemble real reading scenarios and cover a large number of low-resource languages, so it is often used to test the cross-lingual retrieval capabilities of multilingual embedding models. This article uses this dataset to evaluate recall@1 and recall@10 for Tajik, Kazakh, Uzbek, and Kyrgyz, quantifying performance changes before and after fine-tuning.

The unified model maintains the starting model's recall@10 on Tajik legal retrieval (92.2 vs. 92.4, essentially flat), but recall@1 dropped by 4.4 percentage points. This is a noteworthy trade-off: while expanding language coverage, the model sacrifices some fine-grained ranking ability on Tajik legal texts. For high-precision scenarios like legal retrieval, additional calibration or domain adaptation may be needed.

Another interesting finding is: a model trained on one of the languages can also improve Belebele performance in other languages, but the improvement is smaller than training directly on the target language. This suggests some transferability among Central Asian languages, but not enough to fully replace targeted training.

Comparison with Large Models: Still a Gap, but the Path Is Clear

On Belebele for Kazakh, Uzbek, and Kyrgyz, the unified model still lags behind the 560M XLM-RoBERTa embedding model. Specifically, whichever performs better between multilingual-e5-large-instruct and BGE-M3 leads by 9.4 to 10.2 percentage points in recall@1. The author's reproduced scores for these two released models deviate from published MTEB results by no more than 2.4 percentage points, indicating the comparison is reliable.

In addition, the unified model's recall@1 on Russian and English dropped by 3.0 and 4.2 percentage points, respectively. This is a common cost of multilingual fine-tuning—new language capabilities improve while original language capabilities slightly regress. Fortunately, the replay data provides some buffering, so the decline is not drastic.

XLM-RoBERTa is a multilingual pre-trained language model released by Facebook AI, based on the RoBERTa architecture, trained on large-scale corpora in 100 languages, with a base version (about 270 million parameters) and a large version (about 560 million parameters). In embedding retrieval scenarios, XLM-RoBERTa is often used as an encoder backbone, becoming a multilingual embedding model after contrastive learning fine-tuning. BGE-M3 is a multilingual embedding model released by the Beijing Academy of Artificial Intelligence (BAAI), supporting dense, sparse, and multi-vector retrieval modes, with about 570 million parameters. multilingual-e5-large-instruct is the multilingual instruction-tuned version of Microsoft's E5 series, with about 560 million parameters. These models, due to larger parameter counts and more extensive multilingual training data, typically outperform small models at the 300M level on low-resource language retrieval, but deployment costs are also higher. The comparison in this article aims to measure the trade-off between cost and performance for small models.

Implications for Low-Resource Language Embedding Practice

This work provides a reusable template for retrieval systems in low-resource languages: generate question-passage pairs in the target language using synthetic data, perform lightweight continued training on an existing fine-tuned model, and use replay data to prevent forgetting. The 300M model size means low deployment cost, making it suitable for environments with limited resources.

Of course, it is not a solution that "comprehensively surpasses large models." In precision-sensitive scenarios, 560M-level multilingual embedding models remain the safer choice. But if you need to quickly obtain usable retrieval capabilities in four Central Asian languages while controlling model size and inference cost, Saidzoda-CA-EmbeddingGemma-300M offers a data-backed starting point.

Directions worth watching include: how to close the gap with XLM-RoBERTa embedding models, and how to maintain the precision of Tajik legal recall@1 while expanding language coverage.

#EmbeddingGemma#multilingual embedding#low-resource languages#Central Asian languages#fine-tuning#recall@1#Belebele
Share:

Related articles