Token Merging: A Practical Approach to Faster, Cheaper Multilingual Speech Recognition

Multilingual speech recognition models are becoming the dominant direction in voice technology. Models like OpenAI's Whisper can transcribe a wide range of languages — including low-resource ones that lack dedicated training data — without needing to be trained separately for each language. But this capability comes at an obvious cost: the computational overhead at deployment is substantial. A study published on arXiv (arXiv:2609.13151v1) systematically investigates how Token Merging can help address this efficiency bottleneck.

Whisper's Efficiency Problem

The appeal of models like Whisper lies in their generalization: you don't need a dedicated training pipeline for Swahili, Tamil, or other minority languages — the model delivers usable transcriptions out of the box. This matters especially for low-resource languages, which often lack enough labeled data to support independent training.

The problem is that this universality is built on large model sizes and steep inference costs. When you need to deploy at scale in real-world scenarios — live captioning, voice assistants, audio archiving — the cost of processing every second of audio adds up fast. For budget-constrained applications, this is nearly a showstopper.

Whisper was released by OpenAI in 2022 and uses an encoder-decoder Transformer architecture. Audio is first converted into an 80-dimensional Mel spectrogram, downsampled through convolutional layers, and then fed into the encoder to produce a feature sequence. The decoder then generates text token by token in an autoregressive manner. The model family spans multiple sizes by parameter count: Tiny (39M), Base (74M), Small (244M), Medium (769M), and Large (1.5B). Training data amounts to 680,000 hours of weakly supervised multilingual audio. The encoder's computational burden scales linearly with input audio length, and during inference the sequence length is typically fixed at 1,500 tokens corresponding to a 30-second window — precisely where Token Merging intervenes to compress the sequence.

How Token Merging Speeds Things Up

The core idea behind Token Merging is to dynamically merge redundant features during inference. Once a speech signal is encoded into a sequence of tokens, neighboring or similar tokens often carry a large amount of duplicate information. By identifying and merging these redundant tokens, the sequence length the model has to process is reduced, and computation drops accordingly.

The greatest practical value of this approach is that it requires no retraining. It operates at inference time, working directly on existing models without modifying any weights. That means developers can treat it as a plug-and-play optimization — apply it to an existing Whisper model without investing any additional training resources.

Token Merging (ToMe) was originally proposed by Bolya et al. in 2022 to accelerate image classification inference in Vision Transformers (ViT). Its core mechanism uses a bipartite soft matching algorithm to identify token pairs with the highest cosine similarity within each Transformer layer and merge them into a single token (typically via mean or weighted average). This happens before the attention computation, reducing the sequence length that enters the attention mechanism. Computational complexity decreases with the square of the token count, so speedup grows significantly as the merging ratio increases. The key challenge in adapting this method to speech is that audio features are arranged along the time axis, and adjacent frames are inherently highly correlated — making redundant tokens naturally abundant. In theory, this makes speech a better fit for merging than image tasks, which also helps explain why this study observes low accuracy loss on speech transcription.

Systematic Experiments: 16 Languages, 3 Model Scales

The value of this research lies in the breadth of its evaluation. The authors conducted systematic assessments across the Whisper model family, covering:

• 16 different languages, including multiple low-resource languages
• 3 different model sizes, verifying that the method holds across scales
• Interaction with fine-tuning, specifically testing how Token Merging combines with the DoRA fine-tuning method on low-resource languages

This multi-dimensional experimental design avoids the trap of "only works under one specific configuration," lending the conclusions broader applicability.

Core Finding: Near-Lossless Speedup

The results paint a quite positive picture: across most low-resource languages and all model sizes tested, merging tokens improved computational efficiency with almost no loss in transcription accuracy. In other words, users get what amounts to a near-free performance gain — less compute, with barely any sacrifice in recognition quality.

A particularly important finding is that Token Merging remains effective even after a model has been fine-tuned. The researchers tested its combination with DoRA (a parameter-efficient fine-tuning method) and confirmed that the speedup from token merging holds even when the model has been specifically optimized for low-resource languages. This dispels a potential concern: that optimization techniques might interfere with each other.

DoRA (Weight-Decomposed Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method proposed in 2024. Building on LoRA, it further decomposes pretrained weights into magnitude and direction components, updating each separately. This decomposition allows the model to retain more structural information from the original weights during low-rank adaptation. Experiments show it outperforms standard LoRA on multiple tasks while using a similar number of trainable parameters. In low-resource speech recognition scenarios, DoRA's value lies in its ability to significantly improve recognition accuracy on a target language using only a small amount of data, without needing to fine-tune all parameters from scratch. This study tested DoRA in combination with Token Merging specifically to verify whether efficiency optimizations remain applicable after model weights have been customized.

Implications for Real-World Deployment

The practical takeaway of this work is clear — making multilingual speech recognition faster and cheaper to deploy. For engineering teams, it sends several concrete signals:

First, Token Merging is a low-risk optimization. It requires no retraining, has minimal integration cost, and has been validated across multiple languages and model scales.

Second, it is compatible with fine-tuning workflows. Teams can first use a method like DoRA to adapt the model for their target language, then layer on Token Merging for efficiency gains — the two don't conflict.

Third, it is particularly well-suited to low-resource language scenarios. These use cases inherently face data scarcity and tight budgets, and Token Merging maintains accuracy on exactly these languages, making voice applications for more minority languages economically viable.

Conclusion

This research doesn't introduce a new algorithm — it takes the existing Token Merging technique and rigorously validates it in the multilingual speech recognition setting. It answers the questions practitioners care about most: does this method hold up consistently across different languages, different model scales, and after fine-tuning? The answer is largely yes. For any team looking to deploy Whisper-class models at scale, this is a practical optimization worth adding to the toolbox.