MAEP
A training-free expert pruning framework for MoE multimodal large models. It decomposes expert importance by modality and combines Delta-H output changes with low-attention visual token signals for global expert ranking and pruning.
Timeline (last 90 days)
MAEP is a training-free expert pruning framework that requires only a single forward pass to decompose expert importance by modality, compressing a roughly 58 GB MoE multimodal large model into a single 40 GB GPU.
MAEP contains two core components: Delta-H Output (measuring the magnitude of change in the MoE layer's output after removing an expert) and low-attention visual tokens as the visual-side pruning signal.
Low-attention visual tokens as a visual-side importance signal are dataset-stable: the expert importance rankings derived from them remain consistent across different datasets.
In MAEP, text-side importance serves as a protection signal and visual-side importance as a pruning signal, with experts ranked globally across layers.
MAEP was tested on three MoE multimodal models, outperforming modality-agnostic baseline methods on most multimodal benchmarks while remaining competitive on pure text tasks.
With MAEP, expert weight occupancy under BF16 is reduced by about 27 GB, roughly 47% of total memory, allowing all weights of a roughly 58 GB model to fit into a single 40 GB GPU.
The MAEP code is open-sourced at https://github.com/nota-github/MAEP.
MAEP's validation is concentrated on three models and moderate pruning ratios; degradation under extreme compression awaits more practical testing.
All Facts (7)
MAEP is a training-free expert pruning framework that requires only a single forward pass to decompose expert importance by modality, compressing a roughly 58 GB MoE multimodal large model into a single 40 GB GPU.
50%UnverifiedMAEP contains two core components: Delta-H Output (measuring the magnitude of change in the MoE layer's output after removing an expert) and low-attention visual tokens as the visual-side pruning signal.
50%UnverifiedLow-attention visual tokens as a visual-side importance signal are dataset-stable: the expert importance rankings derived from them remain consistent across different datasets.
50%UnverifiedIn MAEP, text-side importance serves as a protection signal and visual-side importance as a pruning signal, with experts ranked globally across layers.
50%UnverifiedMAEP was tested on three MoE multimodal models, outperforming modality-agnostic baseline methods on most multimodal benchmarks while remaining competitive on pure text tasks.
50%UnverifiedWith MAEP, expert weight occupancy under BF16 is reduced by about 27 GB, roughly 47% of total memory, allowing all weights of a roughly 58 GB model to fit into a single 40 GB GPU.
50%UnverifiedMAEP's validation is concentrated on three models and moderate pruning ratios; degradation under extreme compression awaits more practical testing.
50%