Interpretable Acoustic Coordinate System: Rehabilitation Visualization for Japanese Consonant Morae

The study proposes a two-dimensional acoustic coordinate system that provides interpretable dysarthria rehabilitation feedback for Japanese consonant morae.
This article introduces a cutting-edge study on self-rehabilitation for Japanese patients with dysarthria. Because Japanese has mora rhythm and a clear consonant-vowel structure, mora-based feedback is natural, but existing visualization methods are mostly limited to vowels, and consonants are hard to distinguish because transient spectral structure is easily smoothed by generative embeddings. The researchers propose a two-dimensional acoustic coordinate system that maps consonant mora pronunciation deviations onto orthogonal temporal and spectral distance dimensions, forming physically interpretable feedback. Experiments show good geometric separation, a silhouette coefficient above 0.5, origin-based ROC-AUC above 0.86, and ROC-AUC of 0.897 on real dysarthric speech; perturbation tests confirm orthogonality. The system includes consonants in feedback and is expected to generalize to other languages and mobile applications.
Dysarthria reduces speech intelligibility, and patients need fine-grained feedback that captures subtle pronunciation changes for self-rehabilitation. For Japanese speakers, providing feedback in units of morae is a natural choice because Japanese has a distinctive mora rhythm and consonant-vowel structure. However, most existing visualization methods are limited to vowels, and extending distance feedback to consonants is challenging because generative embeddings tend to smooth out transient spectral structure. This article introduces a cutting-edge study that proposes a visualization system for Japanese consonant morae, mapping pronunciation deviations into a two-dimensional space defined by temporal distance and spectral distance, providing physically interpretable feedback for rehabilitation training.

Research Background and Core Problem
Dysarthria is a motor speech disorder caused by neuromuscular damage that impairs control of the articulatory organs, directly affecting speech intelligibility. Traditional rehabilitation training relies on in-person guidance from therapists, but patients lack objective, immediate feedback during daily self-practice and find it difficult to perceive subtle improvements in their pronunciation.
The uniqueness of Japanese lies in its mora rhythm—each mora represents a unit of time, and the consonant-vowel (CV) structure is clear. This structure makes mora-based feedback very intuitive for Japanese speakers. Previous research has attempted to use distance-based visualization methods to provide feedback for vowels, but consonants have remained a challenge. The reason is that consonants often appear as brief transient spectral changes, and commonly used generative embedding methods smooth out these transient structures, reducing the distinguishability between consonants.
Mora is the basic unit used in Japanese to measure syllable duration and is not entirely equivalent to a syllable. For example, in Japanese, 「ん」 (the moraic nasal) and 「っ」 (the geminate consonant) each independently constitute one mora, and the second half of a long vowel also counts as one mora. A typical Japanese word such as 「がっこう」 (school) contains four morae: が, っ, こ, う. Mora rhythm is crucial for Japanese speech perception and pronunciation training because it determines the temporal structure of words. In dysarthria rehabilitation, segmenting speech and providing feedback in units of morae allows patients to precisely perceive which temporal unit deviates, rather than vaguely evaluating the intelligibility of an entire word or sentence. This granularity aligns closely with the intuition of native Japanese speakers and also makes the feedback more actionable.
Generative embedding generally refers to methods that use autoencoders, variational autoencoders, or generative pretrained models to compress high-dimensional speech signals into low-dimensional latent representations. Such methods perform well on steady-state segments such as vowels because vowels have longer stable spectral structure, and embeddings can preserve their discriminative features. However, consonants often last only tens of milliseconds and contain transient spectral events such as bursts and frication, with key information concentrated in extremely short time windows. Generative models tend to minimize reconstruction error during training, prioritizing components with high energy and long duration, thereby smoothing out the transient structure of consonants and causing different consonants to overlap in the embedding space and become difficult to distinguish. This is precisely the core technical obstacle faced when extending distance feedback to consonants in this article.
Method: Two-Dimensional Acoustic Coordinate System
The system proposed by the research team maps pronunciation deviations of consonant morae onto a two-dimensional plane. The two dimensions are:
- Temporal distance: measures the deviation of consonant pronunciation from healthy reference speech in the temporal dimension, such as consonant duration and onset time.
- Spectral distance: measures the deviation of consonants in spectral features, such as spectral energy distribution and formant transitions.
By combining these two orthogonal dimensions, the system can generate a coordinate point for each consonant mora, intuitively showing the difference between the patient's pronunciation and healthy reference pronunciation. This mapping is not only geometrically interpretable but also directly corresponds to physical articulatory properties (time and spectrum), making it easy for therapists and patients to understand.
To validate the system, the researchers conducted a series of experiments, including evaluating the geometric separability of consonant categories, screening tests on real dysarthric speech, and perturbation tests to confirm the orthogonality of the two dimensions.
Experimental Results: Geometric Separation and Screening Performance
Experimental results show that the system performs well on multiple evaluation metrics:
- Geometric separability: Different consonant categories show consistent geometric separation in the two-dimensional space, with a Silhouette Coefficient exceeding 0.5, indicating that both intra-category compactness and inter-category separation are relatively ideal.
- Origin-based ROC-AUC: Using healthy reference as the origin, the ROC-AUC score for screening pronunciation intelligibility is above 0.86, indicating that the system can effectively distinguish normal from abnormal pronunciation.
- Screening on real dysarthric speech: Tested on real dysarthric speech, the system achieved a high ROC-AUC of 0.897 when screening unclear pronunciation, further validating its effectiveness in real-world scenarios.
- Perturbation tests: By artificially perturbing temporal or spectral features, it was confirmed that changes in the two dimensions are orthogonal to each other, that is, shifts in the temporal dimension do not affect judgments in the spectral dimension, and vice versa. This ensures the physical interpretability of the mapping and provides a reliable basis for rehabilitation training.
Together, these results show that the acoustic coordinate system can not only effectively distinguish different consonant morae but also provide meaningful pronunciation deviation feedback for patients with dysarthria.
Silhouette Coefficient is a metric used to evaluate clustering quality, ranging from -1 to 1. For each sample, the average distance to other samples in the same category (compactness a) and the average distance to all samples in the nearest other category (separation b) are calculated, and the sample's silhouette coefficient is (b - a) / max(a, b). The average over all samples is the overall silhouette coefficient. The closer the value is to 1, the more compact the categories are internally and the more separated they are from each other; a value near 0 indicates blurry category boundaries; a negative value indicates that the sample may have been assigned to the wrong category. In this article, a silhouette coefficient above 0.5 indicates that different consonant categories have moderately high clustering quality in the two-dimensional space, with clearly distinguishable category structure.
ROC-AUC, the Area Under the Receiver Operating Characteristic Curve, is a commonly used metric for evaluating binary screening performance. The ROC curve plots the false positive rate on the horizontal axis and the true positive rate on the vertical axis, with AUC ranging from 0.5 to 1: 0.5 is equivalent to random guessing, and 1.0 indicates perfect discrimination. In this article, "origin-based ROC-AUC" means using the position of healthy reference speech in the two-dimensional space as the origin and determining whether pronunciation is clear based on the distance from the patient's pronunciation coordinates to the origin; the larger the distance, the more likely it is abnormal. An AUC above 0.86 means that the system has strong screening ability in distinguishing normal from abnormal pronunciation, and reaching 0.897 on real dysarthric speech further indicates its potential for application in real clinical settings.
Significance and Outlook
This study provides a new visualization tool for self-rehabilitation of dysarthria, specifically addressing the need for fine-grained feedback on Japanese consonant morae. Compared with previous methods that focused only on vowels, this system includes consonants in the feedback scope, covering a more complete set of speech units. Its physically interpretable two-dimensional mapping enables patients to understand the gap between their pronunciation and healthy speech in both temporal and spectral dimensions, allowing them to practice in a more targeted way.
Although the system is currently designed for Japanese, its core idea—mapping pronunciation deviations onto interpretable acoustic dimensions—is expected to generalize to other languages. Future work may include extending it to more consonant types, integrating it into mobile applications, and conducting long-term clinical validation to evaluate its actual impact on rehabilitation outcomes.
One-line Summary
This study proposes a two-dimensional acoustic coordinate system that provides interpretable dysarthria rehabilitation feedback for Japanese consonant morae.
Paragraph Summary
This article introduces a cutting-edge study on self-rehabilitation for Japanese patients with dysarthria. Because Japanese has mora rhythm and a clear consonant-vowel structure, providing feedback in units of morae is relatively natural, but existing visualization methods are mostly limited to vowels, and consonants are difficult to distinguish because transient spectral structure is easily smoothed by generative embeddings. The researchers propose a two-dimensional acoustic coordinate system that maps pronunciation deviations of consonant morae onto two orthogonal dimensions, temporal distance and spectral distance, forming physically interpretable feedback. Experiments show good geometric separation among different consonant categories, with a silhouette coefficient above 0.5, origin-based ROC-AUC above 0.86, and ROC-AUC of 0.897 for screening real dysarthric speech; perturbation tests confirm the orthogonality of the two dimensions. The system includes consonants in the feedback scope and is expected to generalize to other languages and be integrated into mobile applications.
Related articles

Daybreak Red Deployment Guide: A Reference Architecture for Supervised Security Investigation
A breakdown of the Daybreak Red reference architecture: security control design, deployment configuration process, and runtime monitoring for supervised security investigation.

Gemini 4 Neon Tops the Models Engineering Index
Gemini 4 Neon tops the Models Engineering Index, sparking Hacker News debate over ranking credibility and evaluation methods. Here's what it means for developers.

Apple's Self-Developed AI Agent: An Agent That Can Operate Your Mac Is Coming
Apple is reportedly developing an AI agent that can operate a Mac, targeting Meta Muse, Claude, OpenAI Dots, and others. This article analyzes Apple's system-level advantages, the competitive landscape, and likely rollout paths in the agent race.