Technical Terms Getting Transliterated in Multilingual Speech Recognition? Four Solutions Explained

An Overlooked Pain Point in Multilingual Speech Recognition

As on-device multilingual speech-to-text technology becomes widespread, a seemingly minor yet highly challenging engineering problem is surfacing: when users dictate in non-English languages like Korean or Japanese with English technical terms mixed in, models faithfully transliterate these English terms into local scripts based on pronunciation.

On-device speech recognition refers to running the entire speech-to-text inference process locally on the user's device, rather than relying on cloud servers. This trend is driven by the open-sourcing of OpenAI's Whisper model, Google's on-device speech recognition, and the increasing NPU chip capabilities in smartphones. Typical on-device models include lightweight versions of the Whisper series (tiny/base/small), Meta's MMS (Massively Multilingual Speech), and various proprietary compressed models. The advantages of on-device inference include low latency, privacy protection, and offline availability, but the trade-off is limited model parameters, which typically make it harder to handle complex multilingual mixing scenarios like cloud-based large models can.

According to a Reddit developer's real-world testing, this phenomenon is nearly unavoidable. When using on-device multilingual speech recognition, they found:

• GitHub became 기터부 (Korean) / ギットハブ (Japanese)
• Docker became 도커 / ドッカー
• Terms like React, Kafka, and Postgres were all affected

From the model's perspective, this is entirely logical—it's simply faithfully transcribing the sounds it hears. But for developer scenarios requiring precise technical terms, this means any dictated content involving "code-switching" must go through a cleanup pipeline to restore transliterated text back to the actual technical names.

The Root Cause: Code-Switching and Transliteration Mechanisms

Code-switching refers to mixing two or more languages within the same utterance. This is extremely common among technical professionals—developers often describe logic in their native language while using English for framework names, tools, and protocols.

Code-switching is a classic research topic in sociolinguistics, first systematically described by Gumperz in 1982. In computational linguistics, code-switching is subdivided into inter-sentential switching (switching languages at sentence boundaries) and intra-sentential switching (switching within the same sentence). Technical professionals' dictation typically falls into intra-sentential switching, with highly predictable switch points—almost always occurring at proper nouns, tool names, and technical concepts. The challenge for speech recognition systems is that the acoustic model needs to switch phoneme systems in extremely short timeframes, while the language model must determine which writing system the current token should be output in.

When speech recognition models encounter this type of input, they face a dilemma: their acoustic and language models are primarily optimized for the target language (e.g., Japanese, Korean). When encountering English pronunciation, the most natural output is to "spell out" the sound using the local writing system. This isn't a model defect—it's an inevitable result of training data distribution and decoding mechanisms.

Modern end-to-end speech recognition models (like Whisper's encoder-decoder architecture) merge acoustic modeling and language modeling into a unified framework. The encoder converts audio Mel spectrograms into hidden representations, while the decoder generates text token by token in an autoregressive manner. When the model encounters English-pronounced technical terms, the decoder's token vocabulary and training data distribution determine its output tendency: if the training data's Korean audio annotations are primarily Korean characters, the model will tend to spell all sounds it hears using Korean syllables, even when those sounds come from English words. This is essentially the maximum likelihood output of the conditional probability distribution P(text|audio) under specific language conditions.

The truly tricky part is the linguistic differences in the cleanup stage.

Tokenization Traps in Japanese vs. Korean

The original poster ultimately adopted a "correction dictionary keyed by transliterated spellings" approach, but encountered vastly different obstacles between the two languages in practice:

Japanese's problem is "false tokenization." The model inserts spaces in the middle of words, such as outputting GitHub as ギット ハブ (with a space in between). Since Japanese doesn't normally use spaces for word segmentation, these extra spaces can be safely collapsed first to get ギットハブ, which can then be matched against the dictionary.

The Japanese writing system consists of three character sets: kanji, hiragana, and katakana. Katakana (カタカナ) is specifically used for writing loanwords, such as コンピュータ (computer) and インターネット (internet). This writing tradition means that when Japanese speech recognition models encounter foreign word pronunciations, transcribing them into katakana is entirely consistent with linguistic norms. However, for the special category of technical terms, developers typically expect the original English names to be preserved rather than katakana transliterations, because technical documentation and code use the English originals. Japanese's characteristic of not using inter-word spaces (word segmentation relies on morphological analyzers like MeCab) means that model-inserted extra spaces can be safely removed—this is the fundamental reason why Japanese post-processing is relatively simpler.

Korean's problem is exactly the opposite—it uses real inter-word spaces. If you simply strip all spaces, you'll incorrectly merge words that should remain separate, destroying sentence semantics. Korean correction is therefore much more complex than Japanese and cannot use the same space-handling logic.

Korean Hangul (한글) is an alphabetic syllabary where each syllable block is composed of an initial consonant (초성), a vowel (중성), and an optional final consonant (종성). Unlike Japanese, Korean uses real inter-word spaces in its orthography to mark word boundaries, similar to English. Therefore, when speech recognition models output Korean text, spaces carry grammatical meaning—they distinguish structures like "noun + particle" from "compound words." This is why you can't simply collapse spaces as with Japanese: 도커 (Docker) can be safely identified, but if a transliterated word happens to span a real inter-word space, blindly merging would cause semantic errors.

This detail reveals an important principle: Post-processing in multilingual NLP cannot be one-size-fits-all—it must be designed separately for each language's writing system characteristics.

Four Mainstream Solution Approaches

For the challenge of technical term transliteration, several technical approaches have emerged from community discussions, each with its own trade-offs.

Solution 1: Post-Processing Correction Dictionary

This is the lightest and most easily deployable approach. The core idea is maintaining a "transliterated spelling → correct term" mapping table, performing string replacement after speech recognition output.

• Pros: Simple to implement, no model retraining needed, terms can be added/removed anytime, fully controllable
• Cons: Requires manual dictionary maintenance, helpless against new uncatalogued terms, must handle tokenization differences across languages

For vertical scenarios with relatively fixed terminology (e.g., a specific team's tech stack), this is an extremely cost-effective choice.

Solution 2: Contextual Biasing / Hotword Biasing at Decode Time

Contextual biasing or hotword biasing intervenes at the decoding stage. By providing the model with a list of "expected words" (such as a list of common technical terms), it increases the probability weight of these words during decoding, making the model more likely to directly output the correct English original names.

The core idea of this technique is to dynamically boost the output probability of specific vocabulary during beam search decoding, either through Weighted Finite State Transducers (WFST) or attention mechanisms. Specific implementations include: shallow fusion, which adds external language model scores to beam search during decoding; and deep biasing, which encodes the hotword list through a dedicated biasing encoder and interacts with the main model's attention layers. Google's paper "Contextual Speech Recognition with Difficult Negative Training Examples" and NVIDIA's NeMo framework both provide industrial-grade contextual biasing implementations. The key parameter in this approach is the bias weight—setting it too high causes the model to force hotword outputs even on irrelevant audio segments (false recalls), while setting it too low fails to effectively correct transliteration behavior.

• Pros: Solves the problem at the recognition stage, avoiding fragile post-processing matching; good support for dynamic word lists
• Cons: Depends on whether the recognition engine exposes this capability; excessive biasing can cause false recalls

This is currently the recommended direction for industrial-grade speech systems (such as major cloud providers' ASR APIs).

Solution 3: Fine-Tuning the Model on Code-Switched Audio

The most "root-cause" solution is collecting real code-switched audio data and fine-tuning the model to learn to directly output English technical terms in specific contexts rather than transliterating them.

• Pros: Fundamentally improves model capability, most natural results
• Cons: High data collection and annotation costs, significant training resource requirements, may not be applicable to small on-device models

The core challenge of fine-tuning lies in training data construction. An ideal dataset needs to contain a large amount of real code-switched audio with corresponding "expected annotations" (i.e., English terms kept in their original form rather than transliterated). Such data is extremely scarce in public corpora and usually requires targeted collection. Additionally, when fine-tuning small on-device models, catastrophic forgetting is a concern—the model may lose its ability to recognize other common vocabulary while learning to correctly output technical terms.

Solution 4: Accept the Transliteration Status Quo

The final "solution" is to simply accept the transliterated versions and rely on human readers to mentally map them to the correct terms. For informal, purely personal voice notes, this may be the lowest-cost choice—after all, human readers can fully understand that 도커 means Docker.

Engineering Decisions: How to Choose the Right Approach

Solution	Implementation Cost	Effectiveness	Applicable Scenarios
Correction Dictionary	Low	Medium	Fixed term sets, quick deployment
Hotword Biasing	Medium	Relatively High	Controllable recognition engine available
Model Fine-Tuning	High	Highest	Large-scale, long-term products
Accept Status Quo	None	Low	Personal informal scenarios

In practice, these solutions are not mutually exclusive. A mature product often combines multiple approaches: first using hotword biasing to output correct terms during decoding, then using a post-processing dictionary as a safety net to correct remaining errors, and finally considering fine-tuning once sufficient data has been accumulated.

The "Last Mile" of Multilingual Speech Recognition

While the technical term transliteration problem may seem small, it reflects a universal pattern in multilingual AI systems: general-purpose models often require extensive targeted engineering refinement at the boundaries between languages and writing systems. Speech recognition accuracy numbers may look impressive, but in real code-switching scenarios, these "last mile" details determine whether a product is truly usable.

This issue also resonates with broader trends in multilingual AI research. Academic interest in code-switching ASR has been steadily growing in recent years, with dedicated workshops at top conferences like EMNLP and Interspeech discussing multilingual speech processing annually. However, WER (Word Error Rate) metrics reported in research papers often fail to fully reflect actual user experience—a technical term being transliterated may be "approximately correct" at the acoustic level, but its impact on downstream tasks (such as code generation or automated meeting minutes) at the semantic level can be catastrophic.

For developers building multilingual speech applications, it's worth remembering: don't expect a single technical approach to solve all problems. Instead, flexibly combine post-processing dictionaries, decoding biasing, and model fine-tuning based on terminology dynamism, scenario formality, and available resources.

Key Takeaways