RSTGameTranslation: Real-Time OCR Localization + Offline Translation with Local LLMs

A New Option for Raw-Game Players: RSTGameTranslation

Struggling to follow the story in a Japanese game but don't want to wait for a fan translation group? A recently open-sourced tool called RST Game Translation has been drawing attention on Bilibili. Its core capability is straightforward: after you specify a game window or select a region, it uses OCR to recognize foreign text on screen and overlays the Chinese translation back in place. When the on-screen text changes, the translation updates automatically—no need to take repeated screenshots.

According to the Bilibili creator, the tool supports OCR for English, Japanese, Korean, and other languages, with a hotkey to toggle it on and off at any time. Compared with the traditional "screenshot–translate–compare" workflow, this in-place overlay approach is much closer to a "real-time localization" experience.

OCR (Optical Character Recognition) is a technology that detects text regions in an image and converts them into editable text. In gaming scenarios, it typically works in two steps: first locate the position and bounding box of the text on screen, then recognize characters block by block. Because game fonts, outlines, background noise, and dynamic effects vary enormously, OCR is prone to errors on low-contrast small text and stylized fonts—which is a major reason why the accuracy of such tools varies so widely.

Choose Your Translation Engine: Online Key-Free or Local Offline

The flexibility of RSTGameTranslation lies in its translation backend. If you want convenience, you can use an online translation service that requires no API key; if you want to run locally, you can connect to Ollama or LM Studio to load a local LLM for offline translation, eliminating concerns about per-token costs.

This design makes sense for different users: casual players can use the online service out of the box, while privacy-conscious users or those with network restrictions can use local models. The hotkey toggle also ensures the tool won't interfere with gameplay when you need to focus.

Ollama and LM Studio are both local LLM runtimes: the former provides a unified local API as a command-line service, while the latter offers a graphical interface and can load quantized models in formats such as GGUF. Both run model inference entirely on your local GPU or CPU, keeping data on-device—suitable for privacy-sensitive users or those with network restrictions. The trade-off is that they require a certain amount of VRAM/RAM, and small-parameter models typically translate less well than large online models.

Speech Transcription: Translating Dialogue Even Without Subtitles

Even more surprising is its speech transcription capability. The tool uses Whisper-family ASR to recognize character dialogue in games—even if the original game has no subtitles, it can convert speech to text and then translate it, and can even read the translation aloud.

This means its use cases expand from "text on screen" to "audio only." For games that rely heavily on voiced narration, this feature fills a blind spot that OCR cannot cover. However, speech recognition accuracy is affected by accents and background sound effects, so actual performance still needs to be verified by players themselves—the source material does not provide specific performance data.

Whisper is a series of open-source speech recognition models from OpenAI, supporting multilingual transcription, with multiple sizes from tiny to large as well as faster distilled/accelerated variants. It predicts text directly from audio without training language-specific acoustic models, making it suitable for broad general-purpose scenarios. In practice, sample rate, background music, and sound-effect mixing significantly affect recognition quality, and it usually needs to be combined with voice activity detection to segment audio clips.

Beyond Games: Manga Speech Bubbles and Visual Novels

In addition to games, this tool can also handle speech bubble text in manga, Japanese tree signs, visual novels, and other scenarios. Essentially, any foreign text appearing on screen that can be selected and recognized has the potential to be localized in real time.

Packing screen OCR, AI real-time localization, speech transcription, local LLMs, and subtitle overlay into one open-source tool, RSTGameTranslation is quite ambitious. For players who enjoy raw untranslated games, it offers a new approach worth trying. One caveat: the information in this article comes primarily from a single Bilibili user's hands-on share, so details such as OCR accuracy, latency, and compatibility across different games should be tested by users themselves.