PTC-Bias: Improving Rare Word Recognition in Speech LLMs via Phoneme-Level Temporal Competition

Speech large language models (SpeechLLMs) deliver strong performance on general speech recognition tasks, but their accuracy often drops significantly when encountering rare words — proper nouns, technical jargon, personal names, and place names. Contextual biasing is the mainstream approach to this problem: provide the model with a list of candidate words and help it prioritize them. But when the bias list grows to hundreds or thousands of entries, efficiently leveraging that long list becomes a thorny challenge. A new paper on arXiv proposes the PTC-Bias framework — a two-stage solution based on phoneme-level temporal competition.

Why Rare Word Recognition Is Such a Hard Problem

Speech recognition systems are well-trained on high-frequency words, but rare words appear so infrequently in training data that models lack sufficient statistical support, making them prone to recognition errors. Contextual biasing injects prior information through an external word list, which can theoretically improve this situation.

The problem is scalability. When the bias list contains only a few dozen words, the cost of comparing each one is manageable. But once the list expands to 2,000 words, traditional methods either incur heavy computational overhead or, because the candidate space is too large, mistakenly recall many irrelevant words — actually dragging down results that were originally correct. This paper targets exactly that pain point: how to be both accurate and fast with large-scale bias word lists.

Contextual biasing has two main engineering approaches. The first is attention-based: encode candidate words and perform cross-attention with acoustic features, allowing the model to "see" the bias list. The second is prefix-tree (Trie)-based decoding intervention: during beam search, apply extra score rewards to paths that match candidate words. The former is flexible but has computational cost that scales linearly with list size; the latter is lightweight but relies on character- or subword-level matching and is easily confused by words that sound similar but are spelled differently. Traditional ASR systems also use a lighter "hotword weighting" approach that adds a bias term directly to language model scores, but this has limited semantic context awareness and is clearly insufficient in the era of large models. PTC-Bias innovates by pushing the competition granularity down to the phoneme level, sidestepping the ambiguity of subword-level matching, while using temporal alignment to restrict the search to specific frame intervals — enabling efficient filtering without relying on a decoding tree.

PTC-Bias: A Two-Stage Design

The core idea behind PTC-Bias is Phoneme-Level Temporal Competition (PTC), with the entire pipeline split into two stages: retrieval and correction.

Stage 1: PTC Retrieval

During the prefill stage, PTC Retrieval performs frame-synchronous phoneme decoding, letting the pronunciations of candidate words compete against each other along the time axis. Through this temporal competition mechanism, the system filters out a compact shortlist of candidate bias words and simultaneously locates the speech intervals corresponding to those words.

The value of this step is "subtraction first" — rather than dumping the entire 2,000-word list directly into the large model, lightweight phoneme posterior probabilities are used to filter for words that are genuinely likely to appear, drastically compressing the search space for subsequent processing.

Phoneme posterior probabilities are the normalized probability distributions over phonemes that a speech recognition system outputs at each frame, typically from the output layer of a CTC (Connectionist Temporal Classification) or hybrid acoustic model. Frame-synchronous decoding refers to a decoding approach that advances frame by frame along the time axis while maintaining active hypothesis paths — as opposed to batch methods that score an entire utterance at once. Using frame-synchronous decoding to expand candidate words into phoneme sequences essentially projects each word's pronunciation onto the time axis and checks which frame intervals best match the acoustic signal. The computational complexity of this operation depends primarily on the size of the phoneme dictionary and the number of candidate words — not on the SpeechLLM's parameter count — so even with a 2,000-entry list, the impact on overall inference latency is far smaller than an additional forward pass through the large model.

Stage 2: PTC Correction

After the SpeechLLM completes decoding and produces an initial transcript, PTC Correction performs a second round of local competition within the previously located speech intervals, comparing the retrieved candidate words against segments in the transcript that don't match.

This selective correction design is particularly elegant: it specifically targets near-homophones and word-segmentation errors — two of the most common failure modes — and only makes modifications when truly necessary, avoiding disruption to transcript content that was already correct.

A Key Advantage: Zero Additional Forward Passes

A major engineering highlight of PTC-Bias is that both stages share the same set of phoneme posterior probabilities and require no additional SpeechLLM forward passes.

This means neither retrieval nor correction requires re-invoking the large speech model for inference. For deployment scenarios where computational cost is a concern, this is extremely attractive — biasing capability is improved with virtually no additional model inference overhead. This is also what differentiates this method from some approaches that require multiple forward passes or re-decoding.

Experimental Results: Consistent Gains with Large Bias Lists

The paper validates PTC-Bias on the LibriSpeech dataset, covering two different SpeechLLMs with bias lists scaling up to 2,000 words. Results show consistent performance improvements across all settings.

Specifically, using the Prompt-SLAM-ASR-7B model with a 2,000-word bias list:

• On the test-clean set, B-WER (Biased Word Error Rate) decreased 23.4% relative to the CTC-Filter baseline;
• On the test-other set, B-WER decreased 23.9% relative;
• Meanwhile, U-WER (Unbiased Word Error Rate) remained essentially unchanged.

The sharp drop in B-WER alongside stable U-WER validates exactly what the "selective correction" design set out to achieve: improving rare word recognition without sacrificing accuracy on common words. This "fix the weak spot without hurting the strong" property is especially critical for real-world production systems.

B-WER (Biased Word Error Rate) and U-WER (Unbiased Word Error Rate) are split metrics commonly used in contextual biasing research, measuring the recognition error rates for words inside and outside the bias list respectively. These metrics are used instead of overall WER to observe more granularly whether a biasing method genuinely improves recognition of target rare words, while also checking whether it introduces side effects — specifically, whether it incorrectly changes words that were already recognized correctly into phonetically similar words from the bias list (over-triggering). A stable U-WER is the key signal that a biasing approach is "restrained": overly aggressive biasing can lower B-WER while simultaneously raising U-WER, limiting or even reversing overall WER improvement. LibriSpeech, as a standard English read-speech benchmark, has two subsets — test-clean and test-other — representing relatively clean recordings and noisier conditions respectively. Validating on both helps assess the method's robustness.

Industry Implications

As SpeechLLMs increasingly move into production in voice assistants, meeting transcription, and vertical domains like healthcare and legal, accurate recognition of rare words and domain-specific terminology has become an unavoidable requirement. The approach PTC-Bias offers — using lightweight phoneme-level signals to handle retrieval and correction outside the large model — provides a path for large-scale contextual biasing that balances both effectiveness and efficiency.

It's worth noting that current experiments are primarily based on the English LibriSpeech dataset and specific SpeechLLMs. The method's performance on other languages such as Chinese, real-world noisy environments, and even larger bias lists still awaits further validation. But based on published results, this framework demonstrates noteworthy potential in balancing efficiency and accuracy for rare word recognition.