Rethinking Accent Classification: Speaker-Disjoint Evaluation and Cross-Corpus Generalization with DistilHuBERT

Why High Accuracy Can Be an Illusion

Accent classification has repeatedly achieved high accuracy in standard in-corpus evaluations within the speech processing field. But a new study based on DistilHuBERT raises a sharp question: those impressive numbers may significantly overestimate model robustness in real-world deployment scenarios.

The core issue lies in how data is split. When speakers overlap between the training and test sets, the model may actually be learning to "recognize specific speakers" rather than "recognize accent features." This data leakage inflates evaluation results, and once the model encounters speakers it has never seen during training, performance drops sharply.

DistilHuBERT is a lightweight distilled version of HuBERT (Hidden-Unit BERT). HuBERT is a self-supervised speech representation learning model that learns general acoustic feature representations by pre-training on large-scale unlabeled speech data; DistilHuBERT compresses it into a smaller, faster model through knowledge distillation while preserving as much of the original model's representational capacity as possible. In accent classification tasks, researchers typically use it as a feature extractor, fine-tuning a classification head on top of the pre-trained representations. Because the HuBERT family of models excels across a variety of speech downstream tasks, using it as the base model allows the research to focus on methodological issues rather than attributing performance differences to model architecture choices.

File-Level Split vs. Speaker-Disjoint Split

This study uses L2-ARCTIC along with multiple external English accent datasets to systematically compare two evaluation setups: file-level split and speaker-disjoint split.

The difference is this: a file-level split allows different audio files from the same speaker to be distributed across the training and test sets; a speaker-disjoint split strictly ensures that no speaker in the test set has ever appeared during training.

The experimental results reveal a significant performance gap — the difference between in-corpus and external evaluation is substantial. The researchers argue that file-level evaluation effectively overestimates model capability, while speaker-disjoint evaluation provides a more realistic estimate of a model's ability to generalize to unknown speakers. This finding carries an important warning for evaluation methodology in accent classification and, more broadly, speech classification tasks.

Can Training on External Data Solve the Generalization Problem?

An intuitive approach is: if you want to test cross-corpus generalization, just train the model directly on external data. However, the study finds that models trained solely on external data cannot reliably generalize to the L2-ARCTIC dataset.

This demonstrates that significant domain mismatch exists between different corpora. Differences in recording environments, speaker populations, and annotation standards make it difficult for models to maintain stable performance across datasets.

Among all configurations tested, balanced external-data training proved to be the most promising supervised learning setup. By applying class-balanced sampling to external data, the model demonstrated relatively better generalization in cross-corpus testing — though this remains a direction that warrants further exploration.

L2-ARCTIC is a publicly available dataset designed specifically for second-language (L2) English speech research. It contains recordings of speakers from Arabic, Mandarin, Hindi, Korean, Spanish, and Vietnamese language backgrounds reading the same texts, with each speaker providing approximately 1,150 utterances. Because the number of speakers is relatively limited but accent labels are well-defined, it is one of the most commonly used benchmarks in accent classification research. Precisely because of this limited speaker count, training samples under a speaker-disjoint evaluation are even more scarce — a characteristic that amplifies the performance gap between different evaluation setups and makes the study's conclusions more compelling.

The Limitations of Pseudo-Labeling

The study also experimented with pseudo-labeling under a teacher-student framework — using a teacher model to generate labels for unlabeled data, then training a student model on those labels. A common approach is confidence-based filtering, retaining only samples the model is highly certain about.

Experiments show, however, that under domain mismatch conditions, simply relying on confidence-based filtering is insufficient. Models can exhibit high confidence even on incorrectly labeled samples. These "confidently wrong" predictions contaminate the pseudo-labels and undermine the effectiveness of semi-supervised learning. This result reminds researchers that semi-supervised strategies in cross-domain scenarios require more careful design, rather than simply applying a confidence threshold.

The Teacher-Student framework is a classic paradigm in semi-supervised learning: a teacher model (typically already trained on labeled data) makes inferences on a large number of unlabeled samples to generate pseudo-labels; a student model is then trained using these pseudo-labels as supervision. The core assumption is that the teacher model's predictions are reliable enough to provide effective learning signals for the student. Confidence filtering further requires retaining only samples where the teacher model's predicted probability exceeds a certain threshold, to reduce the proportion of noisy labels. However, when the teacher model itself has a systematic domain bias — for example, it has never been exposed to the acoustic conditions of the target domain — its high-confidence predictions tend to cluster around patterns it "knows," while misclassifications on the target domain actually appear with high confidence. This causes the filtering mechanism to fail, allowing large amounts of noise to enter the training data.

Implications for Evaluation Methodology

Taken together, the core contribution of this study is not a stronger model, but rather a critical reflection on the evaluation paradigm for accent classification. It argues that robust accent classification evaluation should incorporate both speaker-disjoint evaluation and cross-corpus testing.

This conclusion is highly relevant to the broader speech research community. Many seemingly strong models may simply be overfitting artifacts of a particular data splitting strategy. Only under more rigorous, real-world-aligned evaluation conditions can the practical value of a model be truly measured. For engineering teams working to deploy accent recognition technology in production, this means validation with unseen speakers and heterogeneous data sources is essential before launch — and that high in-corpus scores should not be mistaken for genuine robustness.