Multilingual Speech Recognition Expansion: Source-Aware Adaptation and Model Merging Strategies

Multilingual ASR systems face a dilemma when expanding to new languages: how can they learn a new language without replaying source-language training data, while still not forgetting the languages they already support? A study from rss, Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR, dives deep into this problem and proposes a solution based on connectors for Speech LLMs.

Background: Catastrophic Forgetting in Connector Adaptation

In connector-based speech LLMs, a trainable connector maps speech representations into a frozen large language model (LLM). The advantage of this architecture is that the LLM itself needs no fine-tuning—training only the connector is enough to achieve speech recognition. However, when the system needs to expand to new languages, target-only fine-tuning of the connector lets the model learn the new language but causes severe degradation on source languages—that is, catastrophic forgetting.

The research team focused on two core questions: how to construct the target-language update, and how to use model merging strategies to preserve source-language capabilities while adapting to new languages. They evaluated a variety of update construction and merging strategies across two scenarios: single-target expansion and multi-target expansion.

Catastrophic Forgetting is a classic problem in continual learning for neural networks: when a model is trained on a new task, parameter updates overwrite or interfere with knowledge learned from previous tasks, causing a sharp drop in performance on the old tasks. In connector-based speech LLMs, because the LLM itself is frozen and only the connector is trainable, forgetting is concentrated in the connector parameters—fine-tuning on a new language substantially changes how the connector maps source-language speech features. Common mitigation strategies include parameter regularization (e.g., EWC), data replay (which requires source data), parameter isolation, and model merging. The source-data-free path explored in this paper falls under parameter isolation and merging: obtain the target-language update separately, then merge it with the source model to restore source-language capabilities.

Connector is the intermediate module linking the speech encoder to the frozen LLM, typically a lightweight adapter (such as a linear layer, MLP, or Q-Former) responsible for mapping the acoustic representations output by the speech encoder into an embedding sequence the LLM can understand. Because the LLM parameters are completely frozen, the connector becomes the only trainable part; its parameter count is far smaller than the LLM's, so fine-tuning is cheap and unlikely to damage the LLM's semantic capabilities. But in multilingual expansion, the connector's capacity is limited, and learning a new language easily crowds out the mapping space for source languages—this is exactly the architectural context in which this paper chooses the connector as the target for adaptation and merging.

Single-Target Expansion: SSU-Wanda+TIES Performs Best

In the single-target expansion scenario, the researchers compared three methods: target-only fine-tuning, standard TIES merging, and SSU-Wanda+TIES. The results show that although target-only fine-tuning and standard TIES merging can learn the new language, both cause severe source-language forgetting. SSU-Wanda+TIES achieved the best balance between source-language retention and target-language acquisition.

SSU-Wanda+TIES combines two techniques: SSU-Wanda selects and protects parameters important for source languages, while TIES reduces interference by merging multiple updates. This combined strategy effectively mitigates forgetting while preserving the learning effect for the new language.

TIES (Trim, Elect Sign, Merge) is a model merging method used to merge multiple fine-tuning updates back into a base model without significant performance loss. Its pipeline has three steps: first, trim each update vector, keeping only the parameters with the largest magnitudes to reduce noise; then elect signs for conflicting parameters, deciding the final sign by majority update direction; finally, merge only the parameters consistent with the elected signs. TIES effectively reduces parameter interference among multiple updates, and in this paper it is used to merge the source-language-protected update with the target-language update.

Wanda is a sparsification/importance scoring method based on weight and activation magnitudes, measuring parameter importance via |weight| × ||activation||. The SSU-Wanda in this paper builds on it by incorporating source-language (Source-Specific Update) information to identify parameters critical to source-language performance and protect them, thereby prioritizing source-language knowledge during merging. The two combine to form the SSU-Wanda+TIES strategy.

Multi-Target Expansion: Update Merging Is Stable, Recursive Expansion Depends on Order

In the multi-target expansion scenario, the researchers found that update merging provides stable source-language retention, whereas recursive expansion can improve target-language adaptation under favorable orders but is highly order-sensitive. This means that when multiple languages need to be expanded at once, merging strategies are more reliable than recursive strategies; recursive strategies have greater potential but require carefully designed language expansion orders.

This finding reveals a key trade-off in multilingual ASR expansion: the interplay among target-language acquisition, source-language retention, and target-language updates. The results show that no single strategy wins in all scenarios; in practice, deployment requires choosing based on specific needs (such as the number of languages and the importance of order).

Update Merging and Recursive Expansion are two strategies in the multi-target scenario. Update merging means computing each target language's parameter update relative to the source model independently, then merging them into the source model all at once or in batches; the language updates do not interfere with each other, so source-language retention is stable. Recursive expansion means adding languages one by one in sequence: after learning each language, the result serves as the starting point for learning the next, so knowledge of earlier languages is overwritten or interfered with by later updates, making it highly sensitive to language order—favorable orders can leverage positive transfer to improve results, while unfavorable orders exacerbate forgetting. This distinction explains the paper's conclusion that "merging is stable, recursion is sensitive."

Implications and Outlook

This study offers practical guidance for expanding LLM-based multilingual ASR systems. For developers who want to add new languages without affecting existing language performance, SSU-Wanda+TIES is a strategy worth trying; for cases requiring simultaneous expansion to multiple languages, update merging is more robust than recursive expansion. In the future, automatically optimizing expansion order and further reducing dependence on source-language data remain directions worth exploring.

Overall, this work highlights the importance of balancing target acquisition, source retention, and update interactions in multilingual ASR expansion, offering new ideas for continual learning in speech LLMs.

One-line Summary

The study proposes the SSU-Wanda+TIES merging strategy to address catastrophic forgetting in multilingual speech LLM expansion.

Paragraph Summary

This paper investigates catastrophic forgetting in connector-based speech LLM multilingual expansion: fine-tuning only the connector to learn a new language causes severe degradation on source languages. The authors systematically evaluate various update construction and model merging strategies, finding in single-target expansion that SSU-Wanda+TIES achieves the best balance between source-language retention and target-language acquisition; in multi-target expansion, they find that update merging stably preserves source languages, whereas recursive expansion may improve target adaptation but is highly sensitive to language order. The study reveals trade-offs among target acquisition, source retention, and update interactions, offering practical guidance for source-data-free continual expansion of multilingual ASR.