Pseudo-Labels as Rewards, Not Targets: How GRPO Improves Semi-Supervised ASR Robustness

In domain adaptation scenarios where only unlabeled audio is available, pseudo-labeling is the mainstream approach: first transcribe audio with a strong teacher model, then fine-tune a student model on these transcriptions. But supervised fine-tuning (SFT) treats every pseudo-label token as a hard target, so the student model inherits the teacher's errors verbatim. A new study proposes using the same pseudo-labels as reward references for Group Relative Policy Optimization (GRPO) rather than as training targets, significantly improving robustness to label noise.

Key Finding: GRPO Achieves 29.3%–56.2% Relative WER Improvement in the Banking Domain

The study conducted comparative experiments in the banking domain: using pseudo-labels generated by Whisper, GRPO achieved 29.3% to 56.2% relative WER (word error rate) improvement over SFT. This advantage is no accident—it holds consistently across four Whisper model sizes, four teacher model families, three domains, and four languages.

More critically, the difference lies in error transfer rates: SFT reproduced the teacher's errors almost 100% verbatim, while GRPO transferred only 27% of teacher errors. This means GRPO effectively filters out noise in pseudo-labels, whereas SFT cements it into the student model.

Word Error Rate (WER) is the most commonly used evaluation metric in automatic speech recognition (ASR), calculated as follows: after aligning the system transcription with the human reference transcription, sum the number of substitution, deletion, and insertion errors, then divide by the total number of words in the reference transcription. Lower WER indicates more accurate recognition.

The "relative WER improvement" mentioned in the article refers to the percentage reduction in WER relative to a baseline system (here, the SFT-fine-tuned student model), calculated as: (WER_SFT - WER_GRPO) / WER_SFT × 100%. For example, if SFT's WER is 10%, a 29.3% relative improvement means GRPO's WER drops to approximately 7.07%. The benefit of relative improvement is that it facilitates comparison across baselines of different difficulty, but readers should note that its absolute value may carry different meanings depending on the baseline WER level. The banking domain typically has higher overall WER than general domains due to a large number of technical terms, numbers, and proper nouns, so the 29.3%–56.2% relative improvement has practical engineering value in this scenario.

Why GRPO Is More Noise-Resistant: An Explanation via Rollout Statistics

The study reveals the underlying mechanism through rollout statistics. In on-policy rollouts, erroneous pseudo-label words rarely appear, so group-relative advantages consistently align with clean transcriptions. In short, GRPO leverages relative comparisons between the model's own sampling behavior and the pseudo-label reward; erroneous labels naturally lack support in sampling and struggle to obtain positive advantages.

Conversely, SFT forces the model to incorporate erroneous pseudo-labels into the policy's support set, which ironically destroys the reward variance that GRPO depends on—once erroneous targets are forcibly fitted, the model loses its ability to distinguish right from wrong. This explains why "the same pseudo-labels, used differently, produce vastly different results."

GRPO (Group Relative Policy Optimization) is a reinforcement learning optimization algorithm that requires no separate value network (critic), originally proposed by the DeepSeek^Ad team for training reasoning capabilities in large language models. Its core idea is: for the same input, sample a group of multiple outputs (rollouts), use the mean reward of outputs within the group as the baseline, compute the relative advantage of each output (advantage = that output's reward - group mean reward), and update the policy accordingly.

Unlike PPO, which requires training a value model of the same scale as the policy to estimate the baseline, GRPO replaces the baseline with intra-group statistics, significantly reducing memory and computational overhead. In this study, the "reward" comes from pseudo-labels: the more consistent the student model's sampled transcription of unlabeled audio is with the teacher's pseudo-label, the higher the reward. The key point is that advantage is the result of relative comparison within a group—if a particular erroneous pseudo-label word rarely appears across multiple samples, then samples containing that word will not receive positive advantage, and the model naturally will not fit that error. This is precisely the algorithmic root of GRPO's noise resistance.

Methodological Implications: A Paradigm Shift in Semi-Supervised ASR

This study conveys a clear signal: the value of pseudo-labels lies not in being imitated as "standard answers" but in serving as "evaluation signals" that guide model exploration. For domain adaptation tasks lacking transcribed data, rather than carefully cleaning pseudo-labels, it is better to change how they are used—employing group-relative policy optimization methods like GRPO to let the model make relative comparisons within its own sampling distribution, thereby naturally avoiding teacher errors.

This idea may also generalize to other semi-supervised sequence generation tasks, such as machine translation or speech translation: as long as a usable strong teacher model and large amounts of unlabeled data exist, converting pseudo-labels from targets to reward references is worth trying.

On-policy rollout refers to sampling and generating sequences under the distribution of the current policy (i.e., the student model itself), rather than using offline data produced by a fixed dataset or an old policy. When training an ASR model with GRPO, for each unlabeled audio, the student model autoregressively samples multiple transcription candidates (rollouts), which reflect the output distribution the model currently "considers" reasonable.

The significance of on-policy sampling is: if the teacher's pseudo-label contains an erroneous word that the student model itself rarely generates at the current training stage, then rollouts containing that word will be at a disadvantage within the group and will not receive positive gradients. In contrast, SFT directly performs maximum likelihood estimation with pseudo-labels as targets, which is equivalent to forcibly pushing erroneous words into the model's support set. Therefore, on-policy rollout naturally constitutes a "noise filter," whose filtering capability depends on the degree to which the student model's own distribution deviates from teacher errors—this is also the mechanistic explanation for why GRPO transfers only 27% of teacher errors.

Experimental Coverage and Generalizability

The robustness of the study is reflected in the breadth of its experimental design: four Whisper sizes verify insensitivity to model scale, four teacher families rule out single-teacher bias, and three domains and four languages demonstrate cross-domain and cross-lingual effectiveness. The 29.3%–56.2% relative WER improvement in the banking domain is the most striking data point, but more important is the consistency of the trend—GRPO outperforms SFT across all these settings.

For engineering practitioners, this means that when building semi-supervised ASR pipelines, you can retain the existing teacher model and pseudo-label generation process, and simply replace the student model's training objective from SFT loss to GRPO reward, potentially achieving substantial robustness improvements.

One-line Summary

Using pseudo-labels as GRPO rewards rather than SFT targets significantly improves semi-supervised ASR robustness to teacher noise.

Paragraph Summary

This article introduces a semi-supervised speech recognition domain adaptation study: in scenarios with only pseudo-labels, traditional SFT treats every pseudo-label token as a hard target, causing the student model to inherit nearly 100% of teacher errors; whereas using the same pseudo-labels as GRPO reward references leverages the fact that erroneous words naturally lack sampling support in on-policy rollouts, transferring only 27% of teacher errors. Experiments achieve 29.3%–56.2% relative WER improvement in the banking domain, with consistent trends across four Whisper sizes, four teacher families, three domains, and four languages. The study further points out that SFT's forced fitting of erroneous pseudo-labels destroys the reward variance that GRPO depends on, revealing the paradigm-shift value of "pseudo-labels as evaluation signals rather than standard answers."