[KongchangAI]
· 1 min read· 856 words

560M Tajik Embedding Model Fine-Tuning: Saidzoda-Tajik-E5-Large Delivers Strong Retrieval Results

560M Tajik Embedding Model Fine-Tuning: Saidzoda-Tajik-E5-Large Delivers Strong Retrieval Results

A 560M Tajik embedding model substantially beats the baseline on legal and tax retrieval.

The article introduces Saidzoda-Tajik-E5-Large, a Tajik text embedding model fine-tuned from multilingual-e5-large-instruct (560M parameters), filling the gap of no publicly available Tajik retrieval model. The authors used their own Tajik language model to generate 235,140 question-passage pairs from legal, tax, Wikipedia, and web text, trained for one epoch, selected hard negatives with another embedding model, and mixed in English and Russian data to prevent forgetting. Top-10 retrieval accuracy rose from 80.2% to 93.6% on legal text and from 82.6% to 94.5% on tax documents; on the human-written Belebele it went from 92.2% to 95.2%, but confidence intervals overlap and the gain is unconfirmed. The model ties with the five-times-larger Giga-Embeddings-3B on similar questions. The authors also report failed attempts such as bfloat16 training collapse and ineffective letter substitution, and note that synthetic test scores should be treated as an upper bound.

In document-driven applications, finding the passage that answers a question is the first step in almost every system. Yet for Tajik, no publicly documented retrieval model existed before. We introduce Saidzoda-Tajik-E5-Large, a Tajik text embedding model built on multilingual-e5-large-instruct (560M parameters).

Saidzoda-Tajik-E5-Large project source

Why Tajik Needs Its Own Embedding Model

Tajik uses the Cyrillic alphabet, but its letter system differs from Russian, and general multilingual models often perform poorly on Tajik. Finding the passage that answers a question is the core step of most document systems, and previously no public Tajik retrieval model was available for reference. We decided to build one ourselves, fine-tuning a multilingual base model specifically for Tajik.

A text embedding model maps a piece of text to a fixed-dimensional dense vector, so that semantically similar texts end up close to each other in vector space. At retrieval time, the system encodes the user's question into a vector, computes cosine similarity against pre-encoded document passage vectors, and returns the most relevant passages ranked by score. Such models are typically based on BERT-style encoders and trained with a contrastive learning objective: pulling the vectors of a question and the correct passage closer, and pushing apart those of incorrect passages. multilingual-e5-large-instruct belongs to the E5 family, whose instruct variant requires a task description prefix before the input (e.g., "retrieve relevant passages for the question") to prompt the model to encode according to the retrieval intent. 560M parameters means it can run inference on a single consumer-grade GPU, making it suitable for language scenarios with limited resources.

Training Data and Strategy: Teaching the Model Tajik Retrieval

We used our own Tajik language model to generate questions for passages from Tajik legal and tax documents, Wikipedia, and web text, ultimately building 235,140 question-passage pairs and training for one epoch. To improve discriminative ability, we used another embedding model to select hard negatives and mixed in English and Russian question-passage pairs to prevent the model from forgetting these two languages.

Hard negatives refer to passages that are semantically close to the question but are not the correct answer. Random negatives often differ clearly in topic from the question, making them easy for the model to distinguish and hard to learn fine-grained discrimination from; hard negatives are retrieved by a stronger embedding model and are typically top-ranked but incorrect candidates, forcing the model to learn finer boundaries. In contrastive learning, negative quality directly affects the discriminability of the embedding space. A common approach is to first run a retrieval pass over the corpus with the base model, take the top-k non-positive passages for each question as hard negatives, and then train with a contrastive loss such as InfoNCE. Mixing in English and Russian data is a multilingual anti-forgetting strategy: if fine-tuned only on Tajik, the model may undergo catastrophic forgetting and lose its original cross-lingual ability; mixing in multilingual samples alleviates this degradation.

On legal texts the model had never seen, 93.6% of questions ranked the correct passage in the top ten, versus a baseline of 80.2%; on tax documents, this figure was 94.5%, versus a baseline of 82.6%. These gains show the model has practical value for domain-specific retrieval.

On Belebele (our only human-written test set), the improvement was smaller and unconfirmed: from 92.2% to 95.2%, with overlapping confidence intervals. Russian and English performance was unchanged.

On legal and tax questions styled like the training questions, the model tied with Giga-Embeddings-3B, which is five times its size; on Belebele, the 3B and 8B models scored slightly higher, but only clearly led after we fine-tuned them on our own data.

Belebele is a multilingual reading comprehension benchmark covering over 100 languages, with questions written by native speakers, and is thus regarded as a more credible evaluation source than synthetic data. It was originally designed to measure reading comprehension and is borrowed here to construct a retrieval evaluation: given a question, check whether the correct passage ranks in the top few positions. Overlapping confidence intervals means the statistical uncertainty ranges of the two scores intersect, so the difference cannot be asserted to be real, usually due to a small test sample size. Top-10 retrieval accuracy refers to the proportion of cases where the correct passage appears in the top ten returned results, a common metric for retrieval systems; compared with top-1, it better matches the practical usage of "handing candidates to a downstream reranker or reader."

Failed Attempts and Limitations

We also report failed attempts: training with bfloat16 caused the model to collapse; replacing Tajik letters with similar Russian letters did not help.

Our legal, tax, and Wikipedia test questions came from the same model that generated the training questions, so we treat these scores as an upper bound until questions written by native Tajik speakers become available.

Conclusion

Saidzoda-Tajik-E5-Large is a 560M-parameter Tajik embedding model that substantially surpasses the baseline on legal and tax retrieval tasks and can compete with larger models. Although the improvement on the human-written test set is not yet significant, the model provides a usable starting point for Tajik retrieval.

#Tajik embedding model#Saidzoda-Tajik-E5-Large#multilingual-e5-large-instruct#text retrieval#fine-tuning#hard negatives
Share:

Related articles