OpenPA: A New Multi-Granularity Pronunciation Assessment Framework Without Forced Alignment

From Script Reading to Free Response: A Paradigm Shift in Pronunciation Assessment

Automatic Pronunciation Assessment (APA) is a core component of language learning and spoken language evaluation technology. In recent years, the field has been undergoing a significant paradigm shift — moving away from fixed-script closed-response evaluation toward open-response assessment. The former requires learners to read a pre-given text, allowing the system to directly compare responses against a reference; the latter allows learners to express themselves freely, which better reflects real-world language use.

This shift brings greater practical value, but it also exposes the limitations of traditional evaluation methods. Most existing systems rely on multi-stage pipelines: first performing forced alignment, then extracting acoustic features through independent feature extractors, and finally outputting a score. This architecture lacks flexibility when dealing with different response settings, making it difficult to handle both closed and open scenarios within a single model.

The Core Idea Behind OpenPA

To address these pain points, researchers have proposed OpenPA — a multi-granularity pronunciation assessment framework that requires no forced alignment. Its key innovation lies in directly leveraging Whisper's cross-attention mechanism to jointly predict both word-level and utterance-level pronunciation scores.

Why Eliminating Forced Alignment Matters

Forced alignment is a time-consuming and error-prone step in traditional APA pipelines, requiring each audio frame to be mapped to a corresponding text unit. Any alignment failure or inaccuracy will cascade into downstream scoring errors. OpenPA leverages Whisper's built-in attention mechanism to automatically establish soft correspondences between audio and text during encoding, bypassing the need for explicit alignment. This not only simplifies the system architecture but also improves robustness against non-standard pronunciation, accents, and even disordered speech.

Mainstream forced alignment tools (such as Montreal Forced Aligner and Kaldi) typically rely on acoustic models and pronunciation lexicons to segment continuous speech into time intervals corresponding to individual phonemes or words. While this works well for native speakers with standard pronunciation, it often produces severe alignment drift for L2 learners' mispronunciations, dialectal accents, or pathological speech — the system may forcibly map an incorrectly produced phoneme to the nearest word boundary, causing subsequent pronunciation scores to be based on incorrect temporal correspondences. Whisper's cross-attention matrix naturally encodes soft relevance weights between decoder text tokens and encoder audio frames, reflecting "which audio segment corresponds to which word" without any hard segmentation. OpenPA directly reads these attention weights as implicit evidence of audio-text alignment, avoiding the fragility of explicit alignment while also eliminating the overhead of maintaining a separate pronunciation lexicon.

Mixed-Transcript Adaptation and Adaptive Substitution Weighting

Two other key techniques in OpenPA are mixed-transcript adaptation and adaptive substitution weighting. The former enables a single model to handle heterogeneous transcriptions from both closed-response and open-response settings, meaning no separate models need to be trained for different scenarios. The latter dynamically weights pronunciation substitution errors during scoring, allowing the model to more accurately capture deviations between actual and target pronunciations.

These two designs together ensure OpenPA's versatility — it can flexibly adapt to diverse input conditions within a unified framework, rather than requiring task-specific customization.

In closed-response settings, the reference text (i.e., what the learner is supposed to read) is known, and the model can directly use the standard transcription. In open-response settings, no reference text exists; the system can only use ASR-generated transcriptions, which may contain recognition errors. The core idea of mixed-transcript adaptation is to feed both types of transcription sources simultaneously during training, allowing the model to learn to switch between them rather than developing a biased dependency on either. Adaptive substitution weighting addresses a classic challenge in pronunciation assessment: when a learner substitutes one target phoneme for another (e.g., producing /s/ instead of /θ/), the impact on pronunciation quality varies significantly across different substitutions. Treating all substitutions as equally weighted errors distorts scores. This mechanism dynamically adjusts penalty weights based on the phonetic distance or frequency of substitutions, making the final score more closely aligned with human rater judgments.

Experimental Performance and Generalization

OpenPA is trained solely on the speechocean762 dataset, yet it demonstrates leading overall performance across multiple evaluation conditions.

• In-domain closed-response: In closed-response scenarios with the same distribution as the training data, OpenPA achieves the strongest overall performance among all evaluated systems.
• Simulated open-response: Under simulated open-response conditions, OpenPA maintains its advantage, validating the effectiveness of mixed-transcript adaptation.
• Zero-shot transfer: On the unseen MultiPA dataset, OpenPA achieves zero-shot performance retention, demonstrating that its learned pronunciation assessment capabilities generalize well across datasets.

Perhaps most intriguingly, the researchers also conducted cross-task evaluation on speech from oral cancer patients. Preliminary results suggest that OpenPA's applicability may extend beyond non-native speaker speech to pathological speech assessment in medically relevant contexts — opening up possibilities for applying speech technology in clinical rehabilitation and speech therapy.

speechocean762 is currently the most widely used public benchmark dataset in pronunciation assessment. It contains approximately 5,000 sentence-level recordings from both native and non-native English speakers, annotated with human scores at the phoneme, word, and sentence levels. It is widely used to measure APA systems' ability to predict word-level accuracy and sentence-level fluency. MultiPA is a diverse evaluation set covering multiple languages and accents, commonly used to verify cross-distribution generalization. Achieving "zero-shot performance retention" on MultiPA means OpenPA requires no fine-tuning or annotation on that dataset — it maintains competitive scoring ability based solely on representations learned from speechocean762. This result directly demonstrates the linguistic generalizability of the model's learned features rather than overfitting to a single dataset.

Technical Value and Potential Applications

The significance of OpenPA lies not only in improving metrics on benchmark datasets, but in reconstructing the technical pathway for pronunciation assessment in a more streamlined and unified way. Eliminating forced alignment lowers the deployment barrier, mixed-transcript adaptation improves scenario adaptability, and preliminary cross-task validation hints at the broader methodological applicability.

For language learning applications, this means a single model can support the full pipeline of spoken language assessment — from classroom read-aloud exercises to free conversation. For medical speech analysis, it offers a potentially transferable tool that requires minimal annotation. Of course, the results on oral cancer speech currently represent only a preliminary exploration, and significant further data collection and validation would be needed before clinical deployment.

Overall, OpenPA represents a valuable direction for directly transferring the intrinsic capabilities of large-scale pretrained speech models (such as Whisper) to downstream evaluation tasks. As demand for open-response assessment continues to grow, alignment-free, highly generalizable frameworks like this are poised to become the new technical mainstream.