Mixture-of-Recursions
Mixture-of-Recursions is a neural network architecture paradigm that combines weight sharing with dynamic routing. By executing different numbers of recursions over shared parameters, it achieves sparse computation along the depth dimension, balancing both parameter efficiency and computational efficiency.
Timeline (last 90 days)
MoR-MLLM is built on the Mixture-of-Recursions (MoR) framework.
Mixture-of-Recursions is a neural network architecture paradigm that combines weight sharing with dynamic routing, where the model shares the same set of parameters across different layers.
MoE applies sparsity in the width dimension (which experts to select), while MoR applies sparsity in the depth dimension (how many layers of computation to pass through).
All Facts (3)
MoR-MLLM is built on the Mixture-of-Recursions (MoR) framework.
50%UnverifiedMixture-of-Recursions is a neural network architecture paradigm that combines weight sharing with dynamic routing, where the model shares the same set of parameters across different layers.
50%UnverifiedMoE applies sparsity in the width dimension (which experts to select), while MoR applies sparsity in the depth dimension (how many layers of computation to pass through).
50%