Back-Translation Still Works: Improving LLM Low-Resource Translation with Monolingual Data

Machine translation for low-resource languages has always been a tough nut to crack in NLP. While pretrained large language models (LLMs) already deliver impressive results on high-resource languages, for languages like Odia and Swahili where parallel corpora are scarce, few-shot prompting or fine-tuning on small-scale parallel data often falls short. A recent study offers a counterintuitive conclusion: if you have a certain amount of monolingual data in a low-resource language on hand, the most effective way to use it is not continued pretraining, but rather a return to a classic technique from the pre-LLM era—Back-Translation (BT).

Core Method: Let the Fine-Tuned LLM Generate Its Own Parallel Corpora

The research team's core idea is not complicated. Given an LLM already fine-tuned on a small-scale parallel corpus, along with monolingual data for each language (both source and target), they have the current model translate monolingual sentences into the other language, generating synthetic parallel sentence pairs, and then use these pairs to train the model in reverse. Updates on parallel data alternate with back-translation-generated data to ensure stability throughout the learning process.

The key judgment here is: rather than using monolingual data for continued pretraining to make the model "familiar" with the low-resource language, it's better to directly convert it into translation training signals. Pretraining only lets the model see the form of the language, whereas back-translation lets the model see the mapping relationship between two languages—the latter clearly provides more direct gains for the translation task.

On Odia and Swahili, using only 1000 parallel pairs, the back-translation method outperforms continued pretraining by 7.7 to 16.4 XCOMET-XXL points in in-domain evaluation. To add one more detail: when beam decoding is used for intermediate translation, models trained with back-translation even surpass models using 10x the parallel data.

Continued Pretraining refers to taking an already pretrained large language model and running another round of self-supervised training (typically with a causal language modeling objective) on monolingual corpora from a specific domain or language, so the model builds stronger language modeling capability in that language. Its advantage is that it can leverage massive amounts of unlabeled monolingual data, but the cost is significant compute, and what it learns is only distribution at the level of linguistic form—not directly targeting translation as a cross-lingual mapping task. In contrast, Back-Translation (BT) is a classic technique from the statistical machine translation era: use the current model to translate target-side monolingual text into the source language, generating synthetic parallel sentence pairs, and train jointly with real parallel corpora. The key to BT is directly converting unlabeled monolingual data into translation supervision signals, rather than merely letting the model "see" the language.

XCOMET-XXL is a next-generation neural translation quality evaluation metric based on the COMET framework, using a large multilingual model underneath to model semantic consistency and error spans between translations and reference translations. Compared to metrics based on n-gram overlap such as BLEU, XCOMET is more robust to word order changes and synonym substitutions, making it better suited for evaluating translation quality on low-resource languages—higher values indicate better translation quality.

Beam decoding is a commonly used decoding strategy in sequence generation that keeps the top k candidate sequences (beam width) with the highest probability at each time step, and finally selects the output with the highest overall score. Compared to greedy decoding, beam decoding can explore more candidate paths and typically yields more fluent and accurate translations, at a slightly higher computational cost. In this study, beam decoding is precisely used to generate higher-quality back-translated sentence pairs, thereby improving subsequent training results.

Key Experimental Findings: Why Back-Translation Outperforms Continued Pretraining

Significant Advantage in Data Efficiency

The most direct appeal of the back-translation method lies in data efficiency. The fact that a model with 10x parallel data is beaten by the back-translation approach means that in low-resource scenarios, the quality of synthetic data may matter more than the quantity of data itself. The sentence pairs generated by back-translation come directly from the current model's best translation outputs, naturally focusing on the model's current capability boundary—essentially a form of adaptive data augmentation.

Gains from In-Context Examples

The study also found that in-context examples retrieved from parallel data can further improve back-translation results. This shows that back-translation is not simply replacing human annotation with a model, but can complement prompting techniques. The retrieved examples provide the model with a clearer translation direction, making the generated synthetic sentence pairs higher quality.

In-context examples refer to providing one or several demonstration pairs of "source sentence—target sentence" in the prompt before feeding the sentence to be translated into the model, so that the model can "learn" the translation direction and style of the current task from the context. Unlike fine-tuning, these examples do not change model parameters and are only provided dynamically at inference time. In low-resource translation, examples can significantly reduce the model's ambiguity about language direction or domain, acting as an implicit conditional constraint. In this study, examples are retrieved from parallel corpora and combined with the back-translation generation process, making synthetic sentence pairs better match the real translation distribution, thereby improving the quality of back-translated corpora.

Cross-Lingual Generalizability Validation

The experiments covered five low-resource languages, with Hindi as a high-resource control group, and used two 3B-scale backbone models. Across all low-resource languages, compared to fine-tuning with only parallel data, back-translation improved XCOMET-XXL scores by 16% to 173% in the direction of translating into low-resource languages. Blind human evaluation on Odia and Sanskrit also confirmed the trend of automatic metrics, showing that this is not just a metric game.

Blind human evaluation refers to evaluators scoring or ranking translation quality without knowing which system generated the translation. This evaluation method can effectively avoid potential biases in automatic metrics and better reflect the true quality of translations in dimensions such as semantic faithfulness and fluency. In this study, the results of blind human evaluation were consistent with the trend of the XCOMET-XXL automatic metric, providing stronger credibility for the back-translation method: its gains are not only reflected in a specific automatic metric, but are also recognized by human evaluators.

Analysis of the Source, Quality, and Quantity of Monolingual Data

The paper specifically analyzes three dimensions of monolingual text: source, quality, and quantity. This part of the analysis has direct guiding significance for practical applications. The source of monolingual data affects the quality of synthetic corpora generated by back-translation—web-crawled data and in-domain data will clearly produce different results. The quality dimension involves noise and domain match; low-quality monolingual data may introduce incorrect translation patterns. As for quantity, more monolingual data usually helps, but with diminishing marginal returns.

These analyses reveal a practical principle: the effectiveness of back-translation depends not only on whether monolingual data exists, but also on what kind of monolingual data it is. When resources are limited, prioritizing domain-matched, quality-controlled monolingual text is more effective than blindly expanding data scale.

Implications for Low-Resource Translation Practice

This study conveys several noteworthy signals. Back-translation, a technique widely used in the statistical machine translation era, is not obsolete in the era of large models. When parallel corpora are extremely scarce but monolingual data is still available, back-translation provides a more efficient path than continued pretraining. For teams working on low-resource language translation, this means there is no need to pursue large-scale parallel corpora or expensive continued pretraining—a fine-tuned 3B model plus a reasonable back-translation workflow can bring significant improvements.

The simplicity of the method is also an advantage. The entire process does not require modifying the model architecture or adding an extra pretraining stage; it only requires inserting back-translation generation and alternating training steps into an existing fine-tuning workflow. This low-barrier characteristic makes it easy to adopt into existing translation pipelines.

Of course, the effectiveness of back-translation depends on the quality of the initial parallel corpus and the selection of monolingual data. How to systematically evaluate and filter monolingual data remains a question that needs further exploration.

One-line Summary

Research proves that in low-resource translation, using LLMs for back-translation of monolingual data is more efficient than continued pretraining.

Paragraph Summary

This article introduces a study on machine translation for low-resource languages. The core conclusion is: when parallel corpora are scarce but monolingual data is available, Back-Translation is more effective than continued pretraining. In the study, on an LLM fine-tuned on a small amount of aligned parallel data, the model translates monolingual sentences into each other to generate synthetic sentence pairs and trains alternately with real parallel corpora. On Odia and Swahili, using only 1000 parallel pairs, the back-translation method outperforms continued pretraining by 7.7 to 16.4 XCOMET-XXL points in in-domain evaluation, even surpassing models using 10x the parallel data. The experiments cover five low-resource languages and are validated by blind human evaluation. The study also analyzes the impact of the source, quality, and quantity of monolingual data on back-translation performance, pointing out that domain match and quality matter more than scale.