MoR-MLLM: Building Efficient Multimodal Large Models with a Mixture-of-Recursions Mechanism

Multimodal large language models (MLLMs) have demonstrated remarkable reasoning capabilities across vision and language tasks, yet their massive computational and memory overhead remains the biggest barrier to real-world deployment. A recent paper on arXiv introduces MoR-MLLM, which fundamentally reimagines the computation paradigm for multimodal models through the idea of "per-token adaptive compute allocation."

The Bottleneck of Existing Lightweight Approaches

To reduce costs, the dominant approach in the field has been to swap in a lightweight language backbone. However, the paper argues that such solutions remain fundamentally "compute-intensive" — their sparsity and depth allocation are static, with no ability to flexibly adapt to the semantic complexity of each token.

In other words, whether a token represents a complex region carrying critical visual information or a nearly negligible redundant symbol, the model invests roughly the same amount of computation in both. This one-size-fits-all treatment wastes enormous compute, and it's the core reason why existing small MLLMs struggle to strike an ideal balance between performance and efficiency.

The Core Idea of MoR-MLLM: Per-Token Adaptive Recursion

MoR-MLLM is built on the recently proposed Mixture-of-Recursions (MoR) framework. It introduces an adaptive per-token recursion mechanism that allows the model to dynamically adjust recursion depth:

• Tokens that are more visually or linguistically challenging receive more recursive computation;
• Simpler, more redundant tokens skip unnecessary processing.

This means compute is no longer spread evenly — instead, like human attention, it is focused on the parts that truly "deserve deeper thought." This dynamic depth allocation is a direct breakthrough over existing static sparsity paradigms.

Mixture-of-Recursions (MoR) is a neural network architecture paradigm that combines weight sharing with dynamic routing. The core idea is that different layers of the model share the same set of parameters (i.e., "recursion") rather than maintaining independent weights per layer as in a standard Transformer. On top of this, MoR introduces a routing mechanism that lets different tokens choose how many times to execute the recursion — essentially "looping" through the same set of parameters a different number of times. This bears some resemblance to Mixture-of-Experts (MoE) sparse activation, but while MoE applies sparsity along the width dimension (which experts to activate), MoR applies it along the depth dimension (how many layers of computation to pass through). Weight sharing dramatically reduces parameter count, while dynamic recursion depth allows the model to reallocate saved compute to tokens that truly need it, achieving both parameter efficiency and computational efficiency.

Designed for Stable Training in Multimodal Settings

Stabilizing recursive sparsity training in multimodal environments is far from trivial — imbalanced routing decisions can easily cause model degradation or training collapse. To address this, the paper introduces two key design elements:

Three-Stage MoR-Tuning Strategy

The research team designed a three-stage tuning pipeline (MoR-Tuning) that gradually guides the model to learn appropriate recursive sparsity on multimodal inputs, avoiding instability in the early stages of training.

Entropy-Regularized Loss

To prevent over-concentration of the routing distribution (i.e., the model defaulting to the same depth for all tokens), the paper introduces an entropy-regularized loss that encourages more diverse routing distributions. This ensures the adaptive mechanism genuinely functions rather than "taking the easy way out" by collapsing to a single mode during training.

Routing collapse is a classic challenge in training Mixture-of-Experts and dynamic routing architectures: due to gradient feedback, the router easily falls into a local optimum where all tokens are assigned to the same path, rendering the dynamic routing mechanism useless. Entropy regularization addresses this by adding a penalty term on the entropy of the routing distribution to the loss function — lower entropy means a more concentrated distribution, resulting in a larger regularization penalty, which incentivizes the router to maintain diverse allocation decisions during optimization. This technique is already widely used in the MoE field (e.g., load balancing losses in GShard and Switch Transformer). MoR-MLLM adapts it to the recursive depth routing setting — a textbook example of porting proven stability techniques to a new paradigm.

Experimental Results and Significance

Through extensive experiments, the paper shows that compared to recent state-of-the-art small MLLMs, MoR-MLLM can significantly reduce training memory and computational complexity while maintaining strong performance across a variety of vision-language tasks.

The significance of this result lies in offering a path to efficiency that is distinct from simply "shrinking the model." Compared to static compression, adaptive recursion allows the model to "tailor its effort" at inference time, finding a better trade-off between efficiency and capability. For developers looking to deploy multimodal models on edge devices or in resource-constrained environments, this class of compute-sparse approaches is well worth watching.

Looking Ahead

MoR-MLLM represents a new direction in efficiency optimization for multimodal large models — shifting from "structural compression" toward "dynamic compute allocation." As frameworks like Mixture-of-Recursions mature, future MLLMs may become increasingly intelligent about deciding "where to spend compute," ultimately making high-performance multimodal inference viable for broad real-world deployment. That said, as a newly published research paper, its generalizability and performance at larger model scales still await further validation.