No More Guessing: New Framework for Locating Speech Recognition Errors Lets the System Ask Questions Proactively

Real-time speech recognition in far-field conversational scenarios still makes many errors, and traditional correction methods either rely on users manually editing or require re-speaking the entire sentence. A new paper proposes a framework called \sysname, which attempts to let the ASR system proactively locate potentially erroneous segments during streaming recognition and request the user to repeat only those segments, thereby maximizing recognition accuracy improvement with minimal interaction cost.

Problem Background: Streaming ASR Errors Are Hard to Remediate After the Fact

In far-field, multi-speaker conversational environments, background noise, reverberation, and speaker distance all significantly degrade speech recognition quality. Although streaming ASR can achieve low-latency output, once a word is misrecognized, subsequent language models often solidify the error, causing the entire sentence's semantics to deviate. Existing approaches typically ask the user to repeat the whole sentence or wait until the session ends for unified correction; the former incurs high interaction cost, while the latter cannot meet real-time requirements.

The paper's authors point out that if the system can determine during recognition which segments are "likely misheard," it can precisely request repetition of just that small portion before the user perceives semantic deviation. This controls interruption frequency while avoiding the experience burden of re-speaking the entire sentence.

The main difference between streaming ASR and offline recognition lies in the real-time constraints of the decoding process. Streaming systems typically employ chunk-based decoding strategies: audio is split into short segments, and the partial hypothesis is updated with each received chunk, with the complete transcription determined only at sentence end or when endpoint detection triggers. To reduce latency, this mechanism often cannot utilize complete future context like offline systems, making local acoustic ambiguity more likely to cause early errors. More critically, streaming decoders usually maintain an internal language model or external rescoring module, and already-output tokens are treated as conditional context by subsequent decoding steps. Once a word is erroneously solidified into the hypothesis sequence, the subsequent language model tends to generate semantically coherent but factually incorrect continuations based on it—a phenomenon known in speech recognition as "error propagation" or "hypothesis solidification." Therefore, in streaming scenarios, the earlier erroneous segments are located and corrected, the smaller the cumulative semantic deviation.

\sysname's Core Idea: Using Clean Audio for "Counterfactual" Supervision

The key innovation of \sysname lies in how the supervision signal is constructed. The research team uses clean audio to estimate what the recognizer "would have output" under better acoustic conditions, thereby constructing a counterfactual reference. Specifically, they use pair-HMM to probabilistically align the noisy decoding results with the clean decoding results, obtaining soft supervision labels—that is, the probability that each token would change if acoustic conditions improved.

These soft labels are used to train a "counterfactual gain scorer." This scorer integrates multiple acoustic and linguistic features, outputs token-level scores, and further aggregates them into segment-level scores to determine which spans warrant triggering repetition requests.

The advantage of this design is that the system does not require manual annotation of "where the errors are"; instead, it automatically learns error patterns from the differences between clean and noisy audio, offering strong scalability.

Pair-HMM (paired hidden Markov model) was originally used for forced alignment in speech recognition, with its core being probabilistic alignment of two observation sequences in a state space. In \sysname, the research team treats the token sequence from noisy decoding and the token sequence from clean audio decoding as two observation paths to be aligned, using pair-HMM to compute their correspondence across time steps. After alignment, the two can be compared token by token for consistency, and consistency is converted into soft labels: if a token is highly likely to become another token under clean conditions, its "counterfactual gain" is high, meaning that position warrants requesting user repetition. Compared to hard labels, soft supervision preserves uncertainty in the alignment process, avoiding noisy labels introduced by forced alignment errors. This approach of using classical alignment tools from signal processing to construct training supervision enables the model to learn error patterns without manual annotation of error locations.

Performance Validation: Significant Recall Improvement, 24% Relative WER Reduction

The experimental data presented in the paper is quite convincing. At a precision rate of 70%, \sysname identifies up to 52% more erroneous tokens than the strongest baseline, achieving the best recall performance. This means that at the same false-positive tolerance, the system can capture more segments that genuinely need user repetition.

In end-to-end evaluation, the targeted repetition strategy yields up to a 24% relative reduction in WER. This figure is achieved under budget constraints that simultaneously consider interruption frequency and repeated segment length, indicating that the gains are not obtained by frequently interrupting users but rather achieved with controllable interaction cost.

Additionally, the framework introduces an LLM to process the ASR transcription after user repetition: the LLM only revises the previously selected segments without altering the entire sentence transcription. This "local repair" approach further reduces the risk of introducing new errors.

WER (Word Error Rate) is the most commonly used evaluation metric in speech recognition, calculated as the sum of substituted, inserted, and deleted error words divided by the total number of reference words. The "24% relative WER reduction" mentioned in the paper refers to a proportional reduction in WER relative to a certain baseline system, not an absolute percentage point decrease. In interactive correction scenarios, WER alone is insufficient to reflect user experience because requesting user repetition incurs interruption costs. Therefore, the paper introduces interaction budget constraints in evaluation: there are upper limits on the number of repetition requests the system is allowed to trigger and the length of repeated segments. The targeted repetition strategy means requesting repetition only for high-value segments selected by the scorer, rather than re-speaking the entire sentence. WER reduction under this setting better demonstrates that the system prioritizes correcting key errors affecting semantics within a limited number of interruptions, rather than indiscriminately increasing interaction frequency.

Implications for Voice Interaction Products

From a product perspective, \sysname represents a paradigm shift in interaction: from "the system passively accepting errors" to "the system proactively managing uncertainty." For far-field scenarios such as meeting transcription, in-car voice assistants, and smart speakers, this on-demand clarification mechanism is expected to significantly improve usability.

However, it should also be noted that the framework relies on clean audio as training supervision, and how to obtain or simulate clean speech in actual deployment remains an engineering challenge. Additionally, the trade-off between interruption frequency and recall needs to be tuned according to specific scenarios—for example, meeting scenarios may tolerate interruptions better than in-car scenarios.

Overall, this research provides a pragmatic path for error handling in streaming ASR: rather than guessing what the user said, it's better to politely ask at critical points.

One-line Summary

\sysname uses clean audio to construct counterfactual supervision, enabling segment-level proactive error correction in streaming ASR.

Paragraph Summary

Far-field streaming ASR is prone to errors in noisy and reverberant environments, and once errors solidify, they are difficult to remediate after the fact. The paper proposes the \sysname framework, which uses pair-HMM probabilistic alignment between clean audio and noisy decoding results to generate soft supervision labels, trains a token-level counterfactual gain scorer, and aggregates into segment-level scores to decide when to request user repetition. Experiments show that at 70% precision, the system identifies up to 52% more erroneous tokens than the strongest baseline, and achieves up to 24% relative WER reduction under interaction budget constraints. The framework also introduces an LLM that only performs local revision on selected segments, avoiding new errors introduced by full-sentence rewriting. This research shifts ASR error handling from passive acceptance to proactive uncertainty management, providing a pragmatic path for far-field scenarios such as meeting transcription and in-car voice.