The Noisy Police Audio Challenge: Opportunities and Limits of Pretrained ASR Pseudo-labeling

Speech recognition technology has long matured for clean audio scenarios, but performance tends to drop off a cliff when faced with real-world, interference-laden recordings. A new arXiv paper, Pretrained ASR Pseudo-labeling for Noisy Police Audio, focuses on one of the most demanding application domains — Broadcast Police Communication (BPC) — and systematically evaluates the applicability and limitations of pseudo-labeling methods in extreme-noise environments.

Why Police Audio Is Such a Hard Nut to Crack

Pretrained ASR systems like Whisper and Qwen3-ASR perform well on standard corpora, but struggle badly on high-noise domains like broadcast police communications. This type of audio typically originates from radio transmissions, accompanied by heavy background noise, signal distortion, specialized terminology, and colloquial speech patterns — all of which make accurate transcription extremely difficult for general-purpose models.

The paper argues that accurately transcribing these communications is important for understanding police decision-making processes. Yet high-quality human annotation is costly and hard to scale. This is precisely where pseudo-labeling comes in: it offers an unsupervised optimization path that bypasses the need for expensive human labels. The question is — how effective is this approach in an extreme-noise domain? That remained an open question until now.

The Failure of Internal Confidence Metrics

The central challenge in pseudo-labeling pipelines is deciding which automatically generated transcriptions are high-quality enough to be used for training.

The research team conducted experiments using BPC corpora from Baltimore and Chicago, and confirmed a key finding: existing internal confidence metrics — including log-probabilities and STAR scores — fail to reliably distinguish high-quality from low-quality pseudo-labels in this setting.

This is a sobering result. In clean-speech scenarios, a model's internal confidence often reasonably reflects transcription reliability. But under extreme noise conditions, this self-assessment mechanism breaks down. The model may assign high confidence to incorrect transcriptions while underestimating actually correct ones, rendering confidence-based filtering strategies ineffective.

Log-probability is the logarithm of the predicted probability a language model assigns to each output token, commonly used as a proxy for how "certain" the model is — the higher the log-probability, the more the model believes the output is correct. The STAR score (Speech Token Aggregated Reliability) is a confidence estimation method built on top of this, combining acoustic confidence with language model probability. The failure mechanism for these internal metrics can be understood as follows: under high-noise conditions, the acoustic input is severely degraded, yet the model may still "confidently" generate a linguistically fluent but incorrect transcription — a phenomenon known as "hallucination with high confidence." Because language models tend to output smooth text consistent with their training distribution, noise doesn't necessarily lower the model's probability estimate for an erroneous output; it may actually push the model to rely more on language priors than acoustic evidence. This is precisely why internal numerical scores cannot reliably filter pseudo-labels in noisy domains.

LLM-as-a-Judge: An External Knowledge Filtering Paradigm

In response to the failure of internal metrics, the paper proposes a new external filtering approach — LLM-as-a-judge.

The core idea is to leverage the parametric knowledge encoded in large language models to assess whether a transcription is contextually plausible, discarding results that are semantically implausible. Unlike relying solely on internal model numbers, LLM-based judgment introduces an external lens on semantic coherence.

Experimental results show that the LLM-judge filtering strategy is more aggressive than internal metrics, and significantly reduces the word error rate (WER) of pseudo-label training sets on both the Baltimore and Chicago BPC corpora. This means that by more strictly removing low-quality samples, the overall quality of training data is meaningfully improved.

That said, the paper maintains appropriate restraint: compared to an oracle filter (a hypothetical filter with perfect judgment), the LLM-judge method still falls significantly short. In other words, this is a promising but far-from-perfect direction.

Cross-Model Pseudo-labeling: A Promising New Paradigm

Another contribution of the paper is the proposal of a cross-model pseudo-labeling paradigm: using pseudo-labels generated by one model to fine-tune another.

Applied to the two base models in the study — Whisper and Qwen3-ASR — this cross-training approach breaks the cycle of bias accumulation that can arise from single-model self-iteration. Different models tend to have different error patterns; one model's weaknesses may be another's strengths, and cross-utilizing pseudo-labels could enable complementary gains. The research team identifies this as a promising direction for future pseudo-labeling research.

Pseudo-labeling is a classic semi-supervised learning technique: first train an initial model on available labeled data, then use that model's predictions on unlabeled data as "pseudo labels" to expand the training set and iteratively refine the model. This paradigm has seen widespread success in image classification and standard speech recognition. However, traditional pseudo-labeling suffers from a "confirmation bias" problem: the model tends to reinforce its own existing errors, and repeatedly using the same model's pseudo-labels causes bias to accumulate and amplify over iterations. Cross-model pseudo-labeling directly addresses this flaw — Whisper and Qwen3-ASR differ in architectural design and pretraining data distribution, meaning their error patterns are not highly correlated statistically. Using model A's output to label data for training model B is equivalent to introducing supervision from an independent information source for model B, effectively breaking the closed loop of single-model self-reinforcement.

Implications for Real-World ASR Deployment

The value of this work extends well beyond the specific domain of police audio — it sheds light on the general difficulties that arise when adapting universal ASR models to new domains.

Methodologically, several conclusions stand out: first, don't blindly trust a model's internal confidence scores — in noisy domains, they may be entirely unreliable; second, the semantic knowledge of an external large language model can serve as an effective quality gate; third, multi-model collaborative pseudo-labeling strategies offer new avenues for low-resource, high-noise scenarios.

For any team attempting to deploy pretrained speech models in noisy real-world environments — such as customer service recordings, medical conversations, or industrial settings — these findings offer directly relevant guidance. Of course, the road from reducing WER to truly approaching human annotation quality remains a long one.