Token Merging
Token Merging is a Transformer inference acceleration technique that uses a bipartite soft-matching algorithm to identify and merge redundant tokens with the highest cosine similarity at each Transformer layer. It reduces computational complexity without requiring model retraining and was originally developed to accelerate vision Transformer inference.
Timeline (last 90 days)
A study published on arXiv (arXiv:2609.13151v1) systematically investigates how Token Merging technology can alleviate efficiency bottlenecks in multilingual speech recognition
Token Merging operates at the inference stage, takes effect directly on existing models without retraining, and does not alter model weights
Token Merging (ToMe) was originally proposed by Bolya et al. in 2022 to accelerate inference for image classification with Vision Transformers (ViT)
ToMe uses a bipartite soft matching algorithm to identify token pairs with the highest cosine similarity within each Transformer layer and merges them into a single token, a process that occurs before attention computation
The study evaluated 16 different languages and 3 different model sizes on the Whisper model family
Experimental results show that for most low-resource languages and various model sizes, Token Merging improved computational efficiency with almost no loss in transcription accuracy
The study demonstrates that Token Merging remains effective after the model has been fine-tuned with DoRA, with the two optimization techniques not interfering with each other
Engineering teams can first optimize the model for the target language using DoRA, then stack Token Merging on top to gain efficiency benefits
All Facts (8)
A study published on arXiv (arXiv:2609.13151v1) systematically investigates how Token Merging technology can alleviate efficiency bottlenecks in multilingual speech recognition
50%UnverifiedToken Merging operates at the inference stage, takes effect directly on existing models without retraining, and does not alter model weights
50%UnverifiedToken Merging (ToMe) was originally proposed by Bolya et al. in 2022 to accelerate inference for image classification with Vision Transformers (ViT)
50%UnverifiedToMe uses a bipartite soft matching algorithm to identify token pairs with the highest cosine similarity within each Transformer layer and merges them into a single token, a process that occurs before attention computation
50%UnverifiedThe study evaluated 16 different languages and 3 different model sizes on the Whisper model family
50%UnverifiedExperimental results show that for most low-resource languages and various model sizes, Token Merging improved computational efficiency with almost no loss in transcription accuracy
50%UnverifiedThe study demonstrates that Token Merging remains effective after the model has been fine-tuned with DoRA, with the two optimization techniques not interfering with each other
50%UnverifiedEngineering teams can first optimize the model for the target language using DoRA, then stack Token Merging on top to gain efficiency benefits
50%