Evaluating Code-Switching ASR: Why WER Falls Short and What to Use Instead

Code-Switching: The Hidden Challenge of Speech Recognition

In the practice of multilingual automatic speech recognition (ASR), code-switching is one of the scenarios most likely to expose a tool's weaknesses. Code-switching refers to speakers naturally alternating between two or more languages within the same conversation, or even within a single sentence. For example, a bilingual speaker might say something like "Je vais au meeting demain" — a sentence mixing English and French.

As a linguistic phenomenon, code-switching was first systematically studied by linguists in the 1950s. Far from being a sign of language deficiency, it is actually a highly skilled linguistic strategy employed by bilinguals. Linguistics typically categorizes it into three types: inter-sentential switching (switching languages between sentences), intra-sentential switching (switching within a single sentence), and tag-switching (inserting exclamations or fixed phrases from another language). From a sociolinguistic perspective, code-switching is driven by multiple social factors including topic, interlocutor, and setting. In the context of globalization, the phenomenon is becoming increasingly prevalent — in multilingual cities, multinational corporations, and on social media, mixed-language communication has become a daily norm, making ASR system support for it increasingly urgent.

A Reddit user testing transcription performance on English/French mixed recordings raised a pointed question: Do existing evaluation metrics truly reflect performance in code-switching scenarios? His observation was quite representative — transcription tools fail precisely at "the most critical points," while the traditional Word Error Rate (WER) metric appears too smooth to capture these fatal mistakes.

This question touches on a long-overlooked pain point in speech recognition evaluation: when language boundaries become blurred, how can we fairly measure the quality of a system?

Why WER Fails in Code-Switching Scenarios

The Basic Logic of WER Calculation

Word Error Rate (WER) is the most mainstream evaluation metric in the ASR field, calculated as:

WER = (Substitutions + Deletions + Insertions) / Total words in reference text

WER treats all word errors equally — whether the error involves an inconsequential function word or a key entity word that changes the entire meaning of a sentence, the penalty is the same. This is the core meaning behind the original poster's observation that "WER feels too flat."

WER has been the de facto standard metric in ASR since the 1990s DARPA-funded large-scale speech recognition evaluation programs (such as Hub-4 and Switchboard evaluations). Its calculation is based on the edit distance (Levenshtein distance) algorithm, which uses dynamic programming to find the optimal alignment path between the reference text and the hypothesis text, then counts the total number of substitution, deletion, and insertion operations in the alignment result. WER has endured for decades because it is simple to compute, reproducible, and convenient for cross-system horizontal comparison. However, WER's design implicitly assumes several important conditions: a monolingual environment, a standardized orthographic system, and that all words have relatively uniform importance. These assumptions generally hold in scenarios like pure English news reading, but are frequently violated in real-world multilingual applications.

Three Challenges WER Faces in Code-Switching Scenarios

In mixed-language scenarios, WER's shortcomings are further amplified:

First, insufficient error weighting at language switch points. Code-switching failures tend to concentrate at language transition boundaries. The system might misrecognize a French word as a similarly-sounding English word, or lose an entire phrase at the switch point. But these "critical failures" are indistinguishable from ordinary errors in WER.

There are deep technical reasons why language switch points become disaster zones for ASR. At the acoustic level, speakers switching languages often undergo an abrupt change in phoneme systems — for example, a sudden transition from English's alveolar approximant /ɹ/ to French's uvular fricative /ʁ/. The acoustic model needs to adapt to a completely different pronunciation system within an extremely short time window. At the language model level, traditional n-gram models and monolingually-trained neural language models assign extremely low probability scores when encountering cross-language word sequences, causing the decoder to force-map foreign words to similar-sounding words in the native language. This is why systems often produce errors at switch points that "sound right but mean something completely different."

Second, ambiguity in tokenization and normalization. Different languages have different tokenization rules. French hyphens, contractions (such as l'homme), and English possessive forms introduce inconsistencies during the text normalization stage before WER calculation, leading to distorted scores.

Text normalization is a critical preprocessing step before WER calculation, but it easily introduces systematic bias in multilingual scenarios. Taking English-French mixing as an example: how to handle French contracted articles like "du" (a merger of de+le), elision forms like the apostrophe in "l'homme," and whether hyphenated compound words like "peut-être" should be split into two words — all directly affect word count statistics and alignment results. On the English side, there are questions about whether "don't" should be split into "do not," differences in reading numbers and dates, and so on. Different evaluation toolchains handle these edge cases with inconsistent standards, potentially resulting in significantly different WER values for the same ASR system across different evaluation frameworks — in severe cases, differences can reach 2-5 percentage points.

Third, the absence of language labels. Traditional WER only cares about "whether the word is correct," not "whether the language was correctly identified." But in bilingual scenarios, identifying "which language this word belongs to" is itself part of the task.

Fairer Evaluation Metrics for Code-Switching

To address the issues above, both academia and industry have developed a series of evaluation methods specifically designed for code-switching.

CER and Language-Aware Error Rates

For morphologically complex languages or those with difficult tokenization, Character Error Rate (CER) is sometimes more robust than WER because it bypasses tokenization ambiguity. However, CER also cannot distinguish between error severity levels.

A more targeted approach is to introduce language boundary-aware metrics, such as:

CSWER (Code-Switching WER): Errors near switch points are counted separately, highlighting performance at boundaries.
Language Identification Accuracy (LID Accuracy): Separately evaluates the system's ability to determine the language affiliation of each word/segment.
Matrix / Embedded Language separate statistics: Distinguishes between recognition rates for the "matrix language" and the "embedded language," since the embedded language (typically the less frequently occurring one) is usually harder to recognize.

Semantics-Oriented Evaluation Methods

Since the original poster was concerned about "failures in critical parts," evaluation oriented toward downstream tasks or semantics is more meaningful:

Entity-level evaluation: Separately measures recognition accuracy for key entities such as personal names, place names, and proper nouns.
Weighted WER: Assigns different weights to words of different importance, giving higher penalties for errors on key words.
Downstream task performance: If the transcription is used for translation, summarization, or semantic retrieval, using the final performance of these tasks to infer ASR quality directly.

What Test Sets to Use for Code-Switching ASR

Beyond evaluation metrics, the choice of test set equally determines whether evaluation is fair. Using purely monolingual datasets to test code-switching capability is inherently unfair. Here are commonly used resource directions from the community and research:

Public Code-Switching Corpora

SEAME: A Southeast Asian English/Mandarin mixed corpus and a classic benchmark for code-switching research.
Miami Bangor (English/Spanish): Conversational recordings from real bilingual communities.
ASCEND, TALCS, and other Chinese-English mixed datasets.

The SEAME (South East Asia Mandarin-English) corpus was collected by Nanyang Technological University in Singapore and contains approximately 200 hours of English/Mandarin mixed conversational recordings, covering both interview and daily conversation scenarios. It has become the gold standard for code-switching research because of its extremely high annotation quality — each word carries a language label, switch points are precisely marked, and recordings come from real bilingual communities rather than artificially constructed mixed sentences. Research based on SEAME has spawned multiple specialized code-switching ASR architectures, including language-aware attention mechanisms, dual-encoder structures, and language identification auxiliary loss functions. Research on this corpus has also revealed an important phenomenon: error rates near switch points are typically 2-3 times higher than in non-switching regions, providing data-driven confirmation that language boundaries are indeed a systematic weak point for ASR.

For English-French mixed scenarios, high-quality public corpora are relatively scarce — one of the practical reasons behind the original poster's evaluation difficulties.

How to Build Your Own Code-Switching Test Set

When off-the-shelf corpora are unavailable, building an evaluation set tailored to your own scenario is often more valuable:

Cover realistic switching density: Include both inter-sentential and intra-sentential switching.
Annotate language labels: Label each word or segment with its language affiliation to enable LID accuracy calculation.
Mark critical segments: Separately flag the "critical parts" you truly care about (such as technical terms, personal names) for weighted evaluation.
Preserve accent and noise diversity: Bilingual speakers have widely varying pronunciation characteristics, and the test set should cover as much of this variation as possible.

Current Technical Landscape of Multilingual ASR Models

The most promising technical approach for handling code-switching today is large-scale end-to-end multilingual models. OpenAI's Whisper, trained on 680,000 hours of weakly-supervised multilingual data covering nearly 100 languages, demonstrates some code-switching handling capability. However, its original design detects language segment by segment and then transcribes in monolingual mode, offering limited support for frequent intra-sentential switching — when a single sentence contains more than three language switches, Whisper's performance degrades significantly. Meta's MMS (Massively Multilingual Speech) covers over 1,100 languages but similarly focuses on monolingual scenarios. Google's USM (Universal Speech Model) performs excellently on multiple languages and dialects, but code-switching remains a known weak point.

In recent years, academia has seen the emergence of hybrid CTC/Attention architectures, language embedding-conditioned decoding, and LLM-based post-processing error correction methods that are gradually advancing solutions to this challenge. Particularly noteworthy is the approach of using LLMs for post-correction of ASR output — by having the language model understand code-switching patterns in context and making semantic-level corrections to the acoustic model's preliminary output, this has already shown significant performance improvements in experiments.

Advice for Multilingual ASR Practitioners

Returning to the original poster's predicament, here are several pragmatic suggestions:

Don't rely on a single metric. Report WER, CER, LID accuracy, and entity-level accuracy simultaneously to provide a multi-dimensional reflection of the system's true performance under code-switching.

Align your evaluation with your goals. If what you care about is getting critical information right, adopt weighted or entity-level evaluation rather than being misled by a smooth overall WER.

Test on realistic distributions. Evaluate using data as close as possible to your actual usage scenario (language pairs, switching density, domain terminology) to avoid the "looks great in the lab, fails in production" trap.

Code-switching serves as the litmus test for ASR precisely because it simultaneously challenges acoustic modeling, language modeling, and language identification capabilities. A truly fair test should also measure these three dimensions from multiple angles, rather than using a single smooth number to mask the failures that truly matter.