Training-Free Image Translation Framework FlowTranslator: A New IIMT Paradigm Based on Flow Editing

In-Image Machine Translation (IIMT) aims to replace the text in an image with a translation while preserving the image's visual context. Traditional methods rely on large amounts of paired before-and-after images to train specialized models, whereas FlowTranslator proposes a training-free alternative. It leverages the multilingual text generation capabilities of advanced text-to-image (T2I) diffusion models to achieve high-quality image translation through flow editing.

Core Idea: Transferring T2I Model Capabilities Without Paired Data

Although existing IIMT models perform well, they require massive amounts of paired training images. The authors of FlowTranslator argue that current T2I models already possess usable multilingual text generation capabilities, and this capability can be directly transferred to the IIMT task. The FlowTranslator pipeline is as follows: first, an OCR model recognizes the text in the image; then an LLM generates a translation prompt; finally, the prompt is injected into a T2I-based diffusion model to generate an image with the replaced text. The entire process requires no IIMT-specific training.

The key to this idea is that T2I models have already learned how to generate multilingual text. FlowTranslator simply uses prompt engineering and trajectory control to guide the model to generate text in the target language within a specified region while keeping the rest of the image unchanged.

Text-to-Image Diffusion Model (T2I) is a class of generative models whose core is a gradual denoising process that transforms random noise into an image matching a text prompt. In recent years, with the growth of model scale and training data, T2I models have shown significant progress in multilingual text generation—that is, the ability to generate text in a specific language within an image based on a prompt. This capability relies on the model's learning of character shapes, typographic patterns, and language context. FlowTranslator leverages exactly this property: instead of retraining the model, it uses prompt engineering (designing translation prompts) and trajectory control (intervening in the diffusion process) to guide the model to generate target-language text in a specified region while keeping the rest of the image unchanged. Therefore, understanding the multilingual text generation capability of T2I models is the key to understanding why FlowTranslator requires no paired data.

Trajectory Control: Region Coupling Release and Coupling Expansion

To optimize the accuracy and naturalness of the generated translated text, FlowTranslator introduces two trajectory control techniques.

Region Coupling Release (RCR) partially decouples the target branch noise within the text region during the early stages of the diffusion process, giving the model greater freedom to modify the text region.

Coupling Expansion (CE) amplifies the newly emerged editing displacement in the middle of the trajectory, making the editing effect more pronounced and thorough.

Together, these two control mechanisms ensure that the generated translation is both accurate and naturally integrated into the original image.

The trajectory of a diffusion model refers to the iterative denoising path from pure noise to the generated image. In standard diffusion sampling, each step depends on the noise estimate from the previous step, and the trajectory is jointly determined by the model and the initial noise. The trajectory control techniques in FlowTranslator intervene in this process to achieve local editing. Specifically, Region Coupling Release (RCR) partially decouples the target branch noise within the text region during the early stages, effectively loosening the coupling between that region and the original image and allowing the model to modify the text content more freely; Coupling Expansion (CE) amplifies the newly emerged editing displacement in the middle of the trajectory, strengthening the editing signal and making the translation more pronounced and thorough. Together, these two techniques ensure that the translated text is both accurate and naturally integrated into the original image without retraining the model.

Experimental Results: Surpassing Training Baselines and Performing Strongly in Real-World Scenarios

Under the EasyOCR read-back protocol, FlowTranslator achieved an average PRIM BLEU of 20.6 and COMET of 57.2, significantly outperforming the strong training baseline UniTranslator. On more challenging real-world scene photos (ScenePair, translated into Chinese), it achieved 40.1% exact match accuracy and surpassed existing editing systems—some of which only handle the text rendering stage rather than the full IIMT task.

These results demonstrate that flow-based editing can serve as a powerful, completely training-free IIMT baseline. FlowTranslator provides a new path for the image translation field that does not rely on large-scale paired data, making it particularly suitable for rapid deployment and multilingual scenarios.

PRIM BLEU and COMET are commonly used automatic evaluation metrics in machine translation. PRIM BLEU is based on n-gram precision and measures the lexical overlap between the generated translation and the reference translation; the higher the value, the closer the translation is to the reference. COMET is based on a pretrained multilingual model and evaluates the semantic quality of the translation, being more robust to word order changes and synonym substitutions. The EasyOCR read-back protocol means first using an OCR model to recognize the text in the generated image, then comparing it with the reference translation to evaluate the accuracy of in-image translation. Exact Match Accuracy requires the generated translation to be completely identical to the reference translation, making it a stricter metric. Together, these metrics constitute the quantitative evaluation of FlowTranslator's performance.

Significance and Outlook

The value of FlowTranslator lies in lowering the barrier to IIMT: there is no need to collect and annotate paired images; existing T2I models and LLMs can be used directly to complete translation. This provides a practical tool for content creators, e-commerce, localization services, and other fields that need to quickly process image text. Of course, the method still depends on the accuracy of OCR and LLMs, and how to further improve robustness in complex scenarios is a direction worth attention in the future.

One-line Summary

FlowTranslator proposes a training-free in-image machine translation framework that uses T2I diffusion models and trajectory control to achieve high-quality translation.

Paragraph Summary

FlowTranslator is a training-free in-image machine translation (IIMT) framework that requires no paired training data. It leverages the multilingual text generation capability of text-to-image (T2I) diffusion models, using OCR recognition, LLM-generated translation prompts, and injection into the diffusion model to generate an image with replaced text. To improve translation accuracy and naturalness, the framework introduces two trajectory control techniques: Region Coupling Release and Coupling Expansion. Experiments show that under the EasyOCR read-back protocol, FlowTranslator achieves PRIM BLEU 20.6 and COMET 57.2, significantly outperforming the training baseline UniTranslator; on the task of translating real-world scene photos into Chinese, it achieves 40.1% exact match accuracy, surpassing existing editing systems. This method provides a new path for IIMT that does not rely on large-scale paired data, making it suitable for rapid deployment and multilingual scenarios.