Oracle-Guided Fine-Tuning: Teaching Generative ASR Post-Editing to Select and Rewrite

Large language models are transforming the paradigm of automatic speech recognition (ASR) post-processing. Instead of retraining the recognizer, one can simply use linguistic context to compare N-best candidate hypotheses and rewrite the transcription when necessary, thereby reducing word error rate (WER). But this path is not without its costs—prompt-based error correction alone is quite fragile: it may over-edit when the top-1 is already correct, hallucinate when the reference text is absent from the candidate list, and often requires extremely large models to work effectively.

To address these pain points, a new study proposes an Oracle-guided fine-tuning method that converts N-best lists into supervision signals, training models to learn a hybrid "select or rewrite" strategy.

The Triple Dilemma of Prompt-Based Error Correction

Existing LLM-based ASR post-editing approaches center on placing N-best hypothesis lists into prompts, letting the model make judgments with its linguistic knowledge. This approach has two clear advantages: it requires no modification to the recognizer itself, and no additional training data. But the study points out that its fragility manifests at three levels.

Over-editing is the first risk. When the top-1 of the N-best list is already correct, the model may still rewrite the output out of an "urge to correct," thereby introducing new errors. Hallucination is the second risk. If the reference text is not in the candidate list at all, the model lacks a reliable basis, yet may generate plausible-sounding content from thin air. Model scale dependency is the third constraint—prompt-based approaches typically require very large models to work reliably, which limits deployment flexibility.

N-best lists are the standard output format of ASR systems: during decoding, the recognizer retains the N most probable candidate transcriptions (typically N ranges from 5 to 10) rather than outputting a single result. The value of N-best lists lies in preserving the uncertainty information of the acoustic model and language model at decoding time—the correct transcription is often hidden in a candidate outside the top-1. Traditional ASR systems use N-best lists for rescoring, i.e., re-ranking candidates with a stronger language model; LLM-based post-editing goes further by not only re-ranking but also potentially generating new text not present in the list. Understanding the structure of N-best lists is a prerequisite for understanding this paper's method: the division into correction pool and hard negative pool is essentially defined by the position of the reference text in the N-best list.

Word Error Rate (WER) is the most central evaluation metric in the ASR field, calculated as the sum of substitution (S), deletion (D), and insertion (I) errors divided by the total number of words in the reference text, i.e., WER = (S+D+I)/N. A lower WER indicates that the transcription is closer to the human-annotated reference text. Note that WER is equally sensitive to insertion errors—this is precisely why this paper emphasizes the risk of "over-editing": even if the model corrects an existing substitution error, if it simultaneously introduces new insertions or substitutions, WER may increase rather than decrease. In generative post-editing scenarios, the model faces a decision of "rewrite or not rewrite," and the increase or decrease in WER depends on the net benefit brought by rewriting, rather than a simple comparison of error counts.

Three-Pool Supervision: Turning N-Best Lists into Training Data

The core innovation of this study is that it no longer relies on prompt engineering tricks at inference time, but instead mines supervision signals from N-best lists, shaping model behavior through three carefully designed sample pools.

Correction pool (corrections): The reference text exists in the N-best list but not at the top-1 position. Such samples teach the model to "select"—to identify the correct candidate buried in the list. This is the core material for learning selection ability.

Regularization pool (regularizers): The top-1 itself is correct. The role of such samples is "restraint"—teaching the model to exercise restraint when no modification is needed, reducing unnecessary rewriting.

Hard negative pool (hard negatives): The reference text is completely absent from the list. Such samples are used to "suppress hallucination"—when the correct answer does not exist, the model should avoid fabricating content.

The three pools correspond respectively to the three abilities of selection, restraint, and anti-hallucination, jointly supporting a hybrid select-or-rewrite strategy. This data construction approach provides more effective supervision signals than random subset fine-tuning.

Oracle information in speech recognition refers to the "standard answer" that is unavailable at inference time—namely the reference text (ground truth). During the training phase, researchers can access the reference text to determine which candidates in the N-best list are correct and which are wrong, thereby constructing supervision signals. The "Oracle guidance" in this paper precisely uses this information available only during training to label sample membership: if the reference is in the list but not at top-1, it goes into the correction pool; if the top-1 is already correct, it goes into the regularization pool; if the reference is completely absent, it goes into the hard negative pool. This approach draws on the supervision paradigms of "quality estimation" and "selective rewriting" in machine translation, with the core being the transformation of Oracle signals unavailable at inference time into supervision labels at training time.

Scale Effects and Dataset Discrepancies

The experimental results reveal an interesting phenomenon. On proprietary datasets, starting from approximately 3B parameters, the gains from fine-tuning are clearest—when the model scale is sufficiently large, the effects of three-pool supervision are fully realized.

However, on the CommonVoice dataset, the conclusions are less clear. The study offers a noteworthy explanation: public read-aloud speech text may overlap with LLM pre-training data, and this overlap can mask true post-editing gains. In other words, the model's "good performance" on CommonVoice may partially stem from memorization rather than genuine post-editing ability. This discrepancy reminds us that when evaluating generative ASR post-editing, the issue of pre-training contamination in datasets cannot be ignored.

Pre-training data contamination refers to the situation where the content of an evaluation dataset has already been seen by the LLM during pre-training, causing the model to potentially achieve good results at test time by relying on memorization rather than genuine generalization ability. As a public crowdsourced read-aloud speech dataset, CommonVoice's text content (mostly Wikipedia sentences and common expressions) is highly likely to appear in LLM training corpora. When candidates in the N-best list closely match pre-training memory, the model can "coincidentally" output correct results even without truly learning post-editing, thereby masking the true gains from fine-tuning. This phenomenon has been widely discussed in generative NLP evaluation, and this paper introduces it to the ASR post-editing field, reminding researchers to set up contamination detection or use proprietary datasets as controls when validating methods on public benchmarks.

Implications for ASR Post-Editing Practice

The value of this work lies not only in proposing a fine-tuning method, but also in redefining the framework of the problem. The traditional approach views LLM post-editing as a prompt engineering problem at inference time, whereas this study transforms it into a supervised learning problem—using oracle information to guide the model in learning when to select and when to rewrite.

For practical systems, this means two things. First, the value of constructing N-best lists is further amplified: they are not only inputs at inference time, but also sources of supervision at training time. Second, the threshold effect of model scale (effective from approximately 3B) provides a reference for deployment decisions, while also suggesting that smaller models may need other compensatory measures. The ambiguous results on CommonVoice serve as a reminder: when validating ASR post-editing methods on public benchmarks, one must be wary of inflated performance caused by pre-training data overlap.

One-line Summary

Oracle-guided fine-tuning converts N-best lists into three-pool supervision signals, training ASR post-editing models to learn when to select or rewrite.

Paragraph Summary

This paper addresses the triple dilemma of over-editing, hallucination, and model scale dependency in LLM-based ASR post-editing by proposing an Oracle-guided fine-tuning method. The method divides N-best lists according to the position of the reference text into a correction pool, a regularization pool, and a hard negative pool, respectively training the model's selection, restraint, and anti-hallucination abilities to form a hybrid select-or-rewrite strategy. Experiments show that on proprietary datasets, gains become evident from approximately 3B parameters, while conclusions on CommonVoice are unclear, possibly due to overlap between public text and LLM pre-training data. The study reframes ASR post-editing from inference-time prompt engineering to a supervised learning problem, emphasizing the value of N-best lists as a source of training supervision, and cautioning against pre-training contamination during evaluation.