MAEP: Modality-Aware Expert Pruning Packs MoE Multimodal LLMs into a Single 40GB GPU

Multimodal large models are rapidly expanding in capability, but their VRAM bills are ballooning in tandem. The MoE (Mixture-of-Experts) architecture was designed to control compute while scaling parameter counts, yet expert parameters remain the overwhelming majority of memory usage. A new paper proposes the MAEP framework, which approaches expert pruning from a previously overlooked angle—modality—and, without any retraining, compresses a ~58 GB model into a single 40 GB GPU.

The Structural Contradiction of MoE: Only a Few Experts Activate, but All Must Be Loaded

MoE's core mechanism routes each token to only a handful of experts, keeping compute in check. But at inference time, all expert parameters must reside in VRAM—in modern MoE multimodal LLMs, expert parameters account for over 90% of model size. The paper's example is intuitive: InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B occupy roughly 58 GB total in BF16, of which 54 GB is expert weights.

This means sparse activation saves compute but cannot save memory. For teams wanting to deploy such models on a single card, expert pruning is practically mandatory: simply delete low-contribution experts, keep the model structure unchanged, and reduce both VRAM and some compute.

MoE Routing and Sparse Activation Mechanism

An MoE layer typically consists of a set of parallel feed-forward networks (i.e., "experts") and a gating network (Router). For each input token, the gating network outputs scores for each expert, then selects only the top K experts by score via Top-K (commonly K=1 or 2), and performs a weighted sum of their outputs according to the gating weights. Unselected experts do not participate at all in that token's forward computation—this is the source of "sparse activation": the model's total parameter count can be very large, but each token's actual floating-point operations depend only on the few activated experts. This is precisely why MoE can expand model capacity without significantly increasing per-token compute cost. However, this sparsity applies only to the computational path, not to storage—all expert weights must be callable at any time during inference, so VRAM usage is determined by the total parameter count rather than the activated parameter count, creating the structural contradiction of "saving compute but not memory."

The Blind Spot of Existing Pruning Methods: Treating Vision and Language Tokens Equally

Most existing expert pruning methods aggregate importance by token, uniformly evaluating how "important" each expert is, then globally rank and prune the tail. The problem is that this aggregation erases a key difference: the same expert may play entirely different roles in visual processing versus language processing.

For pure text models this doesn't matter, but for multimodal models, visual tokens and text tokens differ enormously in quantity, information density, and attention distribution. Uniform aggregation lets whichever modality is more numerous or has stronger attention dominate the pruning decision, and experts genuinely useful for the other modality may be mistakenly pruned.

Expert Pruning and Training-Free Pruning

Expert pruning is a form of structured pruning: it directly removes entire experts (along with their weights and routing entries) rather than sparsifying individual weights, so it genuinely reduces VRAM usage and some compute, and requires no special sparse-compute library support. Based on whether retraining is involved, it falls into two categories. Training-based pruning fine-tunes or distills after removing experts to recover performance—usually more effective but costly; training-free pruning relies on one or a few forward passes to estimate each expert's importance and prunes directly by ranking, without updating any parameters. The key to training-free methods is whether the importance estimate is accurate and whether it is sensitive to calibration data—if swapping in a different calibration dataset drastically changes the expert ranking, the pruning decision lacks credibility. MAEP follows this line, replacing simple routing statistics with a metric that quantifies the "consequences of deletion."

MAEP's Two Core Designs

MAEP (Modality-Aware Expert Pruning) is a training-free framework that decomposes expert importance by modality with just a single forward pass. It contains two key components:

Delta-H Output: Measuring "What Happens If This Expert Is Removed"

ΔH is a metric that accounts for redistribution effects, directly measuring the magnitude of change in the MoE layer's output after removing a given expert. Compared with approaches that only look at routing weights or activation frequency, it is closer to the actual consequence of pruning—once an expert is removed, tokens are redistributed to other experts, and the output change is what truly affects model behavior.

Low-Attention Visual Tokens as a Vision-Side Pruning Signal

The paper finds that low-attention visual tokens can serve as an importance signal on the vision side, and that this signal is dataset-stable: across different datasets, the expert importance ranking derived from it remains consistent. This is crucial—the biggest fear for training-free pruning methods is that importance estimates drift with calibration data; a stable ranking means pruning decisions are insensitive to data distribution, which is more trustworthy in engineering terms.

The division of labor between the two signals is also clear: text-side importance serves as a protection signal, vision-side importance as a pruning signal, and experts are globally ranked across layers.

Redundancy of Visual Tokens and Attention Distribution

In multimodal LLMs, an image is split by the vision encoder into a large number of patch tokens (a high-resolution image can produce hundreds to thousands of tokens), often far exceeding text tokens. The information density of these visual tokens is highly uneven: most patches belong to background or texture-repetitive regions and contribute little to the final answer, while a few key regions carry the main semantics. This redundancy manifests in attention maps as a pronounced long-tail distribution—a few visual tokens receive high attention while the rest receive very low attention. Prior work has exploited this property for visual token compression (e.g., filtering by attention to retain key tokens); MAEP further uses it as a proxy signal for expert importance: experts that contribute little on low-attention visual tokens can be considered unimportant for visual processing. This signal is stable because visual redundancy structure is shared across datasets and does not vary drastically with specific sample content.

Experimental Results: Leading on Multimodal Benchmarks, All Weights Fit on a Single Card

The paper tests multimodal and pure-text benchmarks on three MoE multimodal models. Results show that MAEP outperforms modality-agnostic baselines on most multimodal benchmarks while remaining competitive on pure-text tasks—the latter is important, as decomposing importance by modality does not come at the cost of language capability.

The memory benefit is the most compelling part: under BF16, expert weight usage drops by roughly 27 GB, about 47% of total memory, allowing all weights of a ~58 GB model to fit into a single 40 GB GPU. For the deployment side, this effectively brings a model that would otherwise require multiple cards or large VRAM within the threshold of consumer-grade/single-card server hardware.

The code is open-sourced at https://github.com/nota-github/MAEP.

A Brief Observation

MAEP's approach points to a more general question: do compression strategies for multimodal models need to be designed per modality? In the past, pruning, quantization, and distillation mostly inherited assumptions from text models, whereas the redundancy structure and attention distribution of visual tokens are entirely different from text. Explicitly modeling modality differences into importance evaluation is a low-cost, directly beneficial direction. Being training-free also lowers the barrier to adoption—no retraining needed, importance estimation completes in a single forward pass, making it friendlier for adapting existing models.

It is worth noting that the paper's validation is concentrated on three models; the trade-off curve between pruning ratio and performance, as well as degradation under extreme compression, still require more practical testing. But judging from the step from "58 GB to a single 40 GB card," the direction has been proven feasible.

The Trade-off Between Pruning Ratio and Performance

There is an implicit trade-off curve in expert pruning: the less you prune, the smaller the performance loss but the more limited the VRAM benefit; the more you prune, the more VRAM drops but you may hit the model's capability floor. In practice, one typically plots a curve with "pruning ratio" on the x-axis and benchmark score on the y-axis, seeking the inflection point where performance begins to drop steeply as the operating point. Different layers have inconsistent sensitivity to pruning—shallow experts often handle more general feature extraction, while deep experts may be more specialized—so most methods use global ranking rather than layer-wise proportional pruning. Moreover, degradation under extreme compression is often not smooth and may collapse abruptly after a certain threshold, which is precisely why the paper concentrates validation on moderate pruning ratios and leaves extreme cases for follow-up testing.

One-line Summary

MAEP decomposes expert importance by modality, training-free compressing a 58GB MoE multimodal model into a single 40GB GPU.

Paragraph Summary

The article introduces MAEP, a training-free expert pruning framework for compressing MoE multimodal LLMs. Although MoE controls compute through sparse activation, all expert parameters must reside in VRAM at inference; expert weights account for over 90% of model size—for example, InternVL3.5-30B-A3B and similar models use about 58GB in BF16, of which 54GB is expert weights. Existing pruning methods uniformly aggregate token importance, ignoring the differing roles the same expert plays in vision versus language processing. MAEP proposes the Delta-H Output metric to measure output change after removing an expert, and uses low-attention visual tokens as a vision-side pruning signal that is dataset-stable. Experiments on three models show leading multimodal benchmark performance with competitive pure-text capability, reducing expert weights by about 27GB, enabling the model to fit into a single 40GB GPU.