[KongchangAI]
ConceptModality-Aware Expert Pruning / 模态感知专家剪枝

MAEP

A training-free expert pruning framework for MoE multimodal large models. It decomposes expert importance by modality and combines Delta-H output changes with low-attention visual token signals for global expert ranking and pruning.

Timeline (last 90 days)

Nov 5

MAEP is a training-free expert pruning framework that requires only a single forward pass to decompose expert importance by modality, compressing a roughly 58 GB MoE multimodal large model into a single 40 GB GPU.

Unverified50%
Nov 5

MAEP contains two core components: Delta-H Output (measuring the magnitude of change in the MoE layer's output after removing an expert) and low-attention visual tokens as the visual-side pruning signal.

Unverified50%
Nov 5

Low-attention visual tokens as a visual-side importance signal are dataset-stable: the expert importance rankings derived from them remain consistent across different datasets.

Unverified50%
Nov 5

In MAEP, text-side importance serves as a protection signal and visual-side importance as a pruning signal, with experts ranked globally across layers.

Unverified50%
Nov 5

MAEP was tested on three MoE multimodal models, outperforming modality-agnostic baseline methods on most multimodal benchmarks while remaining competitive on pure text tasks.

Unverified50%
Nov 5

With MAEP, expert weight occupancy under BF16 is reduced by about 27 GB, roughly 47% of total memory, allowing all weights of a roughly 58 GB model to fit into a single 40 GB GPU.

Unverified50%
Nov 5

The MAEP code is open-sourced at https://github.com/nota-github/MAEP.

Expired50%
Nov 5

MAEP's validation is concentrated on three models and moderate pruning ratios; degradation under extreme compression awaits more practical testing.

Unverified50%

All Facts (7)

Source Articles