Routed Sparse Autoencoders: Disentangling Linguistic and Paralinguistic Information in Speech

The Entanglement Problem in Speech Representations

Self-supervised speech encoders (such as WavLM and SPEAR) learn powerful representations from large-scale unlabeled speech data and have become foundational for tasks like speech recognition, speaker identification, and emotion analysis. However, these representations suffer from a long-standing issue: linguistic information (what is being said) and paralinguistic information (who is speaking, with what emotion, and in what prosodic style) are compressed into a single shared, entangled representation space.

This entanglement causes real problems in practice. For automatic speech recognition, you want the model to focus solely on linguistic content, undistracted by speaker identity or emotional tone — while emotion recognition requires exactly the opposite. When these two types of information cannot be cleanly separated, downstream tasks tend to interfere with each other, and fine-grained control in applications like privacy protection and voice conversion becomes difficult.

This arXiv study proposes a method that routes these two types of information into distinct representation channels — without retraining the encoder — achieving controllable disentanglement.

Core Method: Routed Sparse Autoencoders

The technical foundation of this work combines a TopK Sparse Autoencoder (SAE) with two additional mechanisms:

Route-Specific Supervision

The model splits the encoder's output features into two "routes" — a linguistic route and a paralinguistic route. Targeted supervision signals guide linguistic information toward the linguistic route, while paralinguistic information (speaker identity, emotion, prosody) flows into the paralinguistic route.

Cross-Factor Adversaries

Supervision alone is insufficient — information may still leak into both routes simultaneously. The research introduces adversarial training to actively suppress information that shouldn't appear in a given route. For example, the linguistic route is trained to "forget" speaker identity and emotional state. This two-directional constraint of "preserve + suppress" is key to achieving clean separation.

Importantly, the entire process operates on frozen encoders — neither SPEAR nor WavLM weights are modified. Only the representations at the output level are reorganized, which significantly lowers the cost of applying this method.

---

Sparse Autoencoders (SAEs) are unsupervised neural network architectures that impose sparsity constraints on hidden layers, forcing the model to reconstruct inputs using only a small number of active neurons, thereby learning more interpretable distributed features. The TopK variant further specifies that only the K neurons with the highest activation values are retained per forward pass, with all others set to zero, enabling precise control over sparsity. SAEs have attracted considerable attention in LLM interpretability research over the past couple of years — researchers have found that applying SAEs to intermediate-layer activations in LLMs can extract individual neuron directions corresponding to specific concepts or linguistic features, known as "interpretable features." This paper transfers that same intuition to the speech domain: the SAE acts as a "feature decomposer," unfolding entangled speech representations into a higher-dimensional sparse space, while the routing mechanism directs features of different semantic categories into their dedicated channels.

Adversarial training for representation disentanglement has classical roots in domain adaptation and fairness learning. The basic framework involves attaching a "discriminator" to the main encoder that specifically tries to predict a factor that shouldn't exist in a given route (e.g., predicting speaker identity from the linguistic route). The main network is then trained to fool this discriminator, effectively erasing that factor's information from the corresponding route during backpropagation. This is analogous to the generator-discriminator dynamic in GANs, but the goal is not to generate realistic samples — it's to remove unwanted information dimensions. The "cross" in "cross-factor adversaries" reflects the fact that adversarial signals simultaneously target multiple paralinguistic factors (speaker, emotion, prosody) rather than a single one, enabling synchronized multi-dimensional suppression.

---

Experimental Results: Does the Separation Actually Work?

The study uses independent probes to measure the strength of each type of information in both routes, yielding highly consistent results:

• Linguistic information remains strong in the linguistic route and is weakened in the paralinguistic route;
• Paralinguistic factors (including speaker identity, emotion, and prosody) are preserved in the paralinguistic route and significantly reduced in the linguistic route.

This demonstrates that the route separation is not superficial — it achieves factor-specific preservation and suppression.

Cross-Dataset Generalization

A compelling finding is that the routing organization learned on LibriSpeech transfers directly to the MSP-Podcast dataset without any retraining on the representation side. LibriSpeech consists primarily of read speech, while MSP-Podcast features emotionally rich podcast conversations — the two differ substantially in style. The persistence of the routing structure suggests the separation captures a fundamental pattern of information organization, rather than overfitting to a specific corpus.

---

LibriSpeech is a large-scale English speech dataset compiled from the LibriVox audiobook project. It features read-aloud speech with minimal emotional variation and relatively clean recording conditions, and is widely used as an ASR benchmark. MSP-Podcast is a natural conversation corpus collected from podcast programs, containing rich emotional expression and real-world noise, and serves as an important benchmark in affective speech analysis. The two datasets differ significantly in recording environment, speaking style, and emotional richness, making a transfer from LibriSpeech to MSP-Podcast a fairly rigorous out-of-domain generalization test. The fact that the routing structure transfers across domains without retraining suggests that the linguistic/paralinguistic boundary learned by this method reflects an intrinsic organizational pattern of the speech signal, not a memorization of any particular corpus style.

---

Feature-Space Intervention and Factor Swapping

The study also conducts representation-level intervention experiments: swapping the factors carried by one route in the feature space. Results show that the swapped factors transfer successfully, while information carried by the unmodified route is largely preserved. This "factor swapping" capability provides a directly actionable handle for downstream applications such as voice conversion, emotion transfer, and speaker anonymization.

Significance and Application Prospects

The value of this work lies in validating the consistency of route-selective separation across multiple dimensions — across encoders (SPEAR and WavLM), across corpora (LibriSpeech and MSP-Podcast), across independent probes, and across representation-level interventions. This multi-angle cross-validation lends credibility to the findings.

From an application standpoint, controllable linguistic/paralinguistic disentanglement enables:

• Privacy protection: Speaker identity can be stripped while preserving linguistic content, enabling voice anonymization;
• Voice conversion and synthesis: Flexible swapping of timbre and emotion by exchanging paralinguistic routes;
• Cleaner downstream tasks: ASR can focus on the linguistic route while emotion recognition focuses on the paralinguistic route, reducing mutual interference.

Sparse autoencoders have shone in LLM interpretability research in recent years. This paper extends that line of thinking to the speech domain, augmenting it with routing supervision and adversarial mechanisms to offer a noteworthy technical path toward interpretable and controllable speech representations.

Conclusion

Speech signals are inherently a dense interweaving of linguistic and paralinguistic information. Cleanly separating them without sacrificing representation quality has long been a core challenge in speech representation learning. This study offers a lightweight yet effective answer through the combination of frozen encoders and routed sparse autoencoders. Its generalization performance across corpora is particularly worthy of deeper exploration in future work.