How Model Compression Amplifies Group Disparities in Whisper Speech Recognition

The Blind Spot in Full-Precision Fairness Audits

Automatic speech recognition (ASR) models typically undergo fairness audits before deployment, evaluating whether their performance is equitable across different demographic groups. But these audits almost always target full-precision models — while what actually gets deployed in production is a compressed version that has been quantized, pruned, or distilled.

A new arXiv paper (arXiv:2609.28739v1) poses a pointed question: Does post-training weight compression redistribute the error burden unevenly across demographic groups? Unlike methods that modify audio signals or feature representations, weight compression directly alters model parameters — yet its impact on fairness has rarely been systematically quantified.

The research team evaluated the Whisper model family across three datasets — Fair-Speech, Common Voice 25, and AfriSpeech-200 — and reached a sobering conclusion: a single snapshot-style full-precision audit simply cannot capture the real burden that compression places on marginalized groups at deployment time.

Pruning: Doubling the Disparity

The study adopts and operationalizes the concept of "temporal taxation" introduced by Choi and Choi. The temporal tax refers to the additional time users must spend correcting transcription errors — groups with more errors carry a heavier tax.

The most striking finding comes from pruning. Applying 50% Wanda pruning to Whisper-large-v3 caused the temporal tax gap between Black/African American and Asian speakers on the Fair-Speech dataset to balloon dramatically: the absolute word error rate (WER) gap between the worst-served and best-served groups grew to more than twice its original size.

The numbers are stark in concrete terms: assuming each transcription error costs 5 seconds to correct, the correction time per minute of audio rose from 30 seconds to 64 seconds — a +111% relative increase.

The researchers also validated the robustness of this finding. The result is independent of the assumed per-error correction cost (i.e., the proportional increase holds regardless of how many seconds are assigned per error), and it holds up under audio quality controls. Even with beam-search decoding as a mitigation strategy, the gap retained a +86% increase, demonstrating that decoding choices can only partially repair the fairness damage inflicted by compression.

Wanda (Pruning by Weights and Activations) is a post-training pruning method that requires no retraining. It determines which parameters to zero out by combining information about weight magnitude and the statistics of input activations. Compared to traditional magnitude pruning, Wanda typically maintains better overall performance at high compression ratios, making it a widely used baseline in large model compression research. A 50% pruning rate means half of the model's weights are set to zero; the model skips these connections at inference time, reducing both computation and storage. While this structural removal may appear to have limited impact on aggregate accuracy metrics, the pruned weights may include parameters critical for processing the speech characteristics of underrepresented groups in the training data — causing error rates for marginalized groups to rise disproportionately.

Quantization: A Catastrophic Loop for West African Accents

If pruning "doubles" the disparity, extreme quantization at edge-device model sizes can trigger outright collapse.

The study found that with INT4 HQQ quantization, catastrophic transcript loops when processing West African accents were amplified by a factor of 5 to 7. Transcript loops refer to a pathological state in which the model falls into repetitive output, generating large amounts of meaningless repeated text — a failure mode particularly common in low-bit quantization.

For accent groups that are already at a recognition disadvantage, this kind of collapse means service quality that is nearly unusable. Crucially, edge devices are precisely where quantized models are most commonly deployed — to run on compute-constrained hardware, aggressive quantization like INT4 is practically mandatory. This means the regions and populations most in need of low-cost AI services may be bearing the most severe quality degradation.

HQQ (Half-Quadratic Quantization) is a post-training quantization method designed for extremely low bit-widths. It uses a half-quadratic optimization framework to minimize quantization error, achieving significantly lower performance loss than standard uniform quantization at INT4 precision. INT4 quantization compresses each weight from the typical 32-bit or 16-bit floating point down to just a 4-bit integer, reducing storage by roughly 8×, enabling large models that would otherwise require servers to run on phones, smart speakers, and other edge devices. However, a 4-bit integer can only represent 16 discrete values — this extreme coarseness severely damages the model's robustness when handling out-of-distribution inputs such as accents that are rare in the training data. Catastrophic transcript loops are a known failure mode of autoregressive speech models like Whisper: when the model lacks sufficient confidence to predict the next token normally, it can fall into a loop of repeatedly generating the same segment, producing hundreds or even thousands of repeated characters. This problem is significantly amplified when low-bit quantization degrades model capacity.

Distillation: The Only Compression Method That Improves Fairness

Not all compression techniques make things worse. The study found that distillation exhibits the opposite trend.

Across 27 evaluation settings (spanning different teacher-student model pairs, precisions, and datasets), distillation narrowed group disparities in 21 of those settings. The handful of cases where it worsened fairness were concentrated in a single model pair.

This finding is illuminating: by having the smaller model learn from the soft label distributions of the larger model, distillation in some sense preserves or even smooths how the model handles different demographic groups — unlike pruning and quantization, which bluntly delete or truncate weights. This suggests that when pursuing model compression, the choice of technique directly determines fairness outcomes.

Knowledge Distillation, introduced by Hinton et al., centers on using a trained large model (the teacher) to guide training of a smaller model (the student). Rather than training solely on hard labels (one-hot ground truth), the student also learns from the teacher's soft labels — the probability distribution the teacher assigns across all possible outputs. These soft labels encode the inter-class similarity information that the teacher has learned; in speech recognition, for instance, the teacher's handling of different accent variants is implicitly conveyed to the student through probability distributions. The mechanism by which distillation improves fairness in this study may lie precisely here: the teacher model's generalization ability across different demographic speech patterns — learned from large amounts of data — is more completely transferred to the student via soft-label supervision, rather than being uniformly truncated as in pruning or quantization, which damage feature representations for marginalized groups disproportionately.

Practical Implications

The core contribution of this work is exposing a structural flaw in current AI fairness evaluation pipelines: a mismatch between what is audited and what is deployed.

Organizations conduct fairness audits on full-precision models, declare them to have "passed," and then deliver compressed versions to users. The compression process systematically offloads extra error burden onto already-marginalized speech communities — Black/African American speakers, West African accent users, and others.

The implications for practitioners are clear:

• Fairness audits must be conducted on the actual compressed models used in deployment, not full-precision baselines
• Pruning and aggressive quantization carry group fairness costs that need explicit evaluation alongside efficiency gains
• Distillation may be a more "fairness-friendly" compression path and deserves prioritization
• Decoding optimizations like beam search can only partially mitigate the problem — they are not a substitute for care in choosing compression techniques

As ASR and large models increasingly migrate to edge devices, "compress-then-deploy" scenarios will only become more common. This research reminds us that efficiency optimization is never a neutral technical decision — it can quietly amplify existing inequalities.