CoPre-MoE: Expert Merging and Forgetting Suppression in Multimodal Continual Instruction Fine-Tuning

CoPre-MoE balances anti-forgetting and parameter efficiency in continual instruction fine-tuning by merging overlapping experts and decomposing private experts.
The article centers on the "learn new without forgetting old" tension in continual instruction fine-tuning of multimodal large language models, noting that mainstream MoE methods allocate independent experts per task, which avoids parameter overwriting but leads to cross-task functional redundancy and linear expert growth. CoPre-MoE proposes two-stage compression: first selectively merging functionally overlapping experts into a fixed-capacity shared expert pool, then decomposing remaining task experts into a shared projection basis and compact task cores placed in a private pool. At the routing level, it freezes historical retrieval keys and applies orthogonal regularization to new keys, enabling dynamic expert composition without task labels. Experiments claim better performance with substantially lower parameter overhead, but trade-offs such as shared pool capacity and regularization strength still require further validation.
The Core Tension of Multimodal Continual Instruction Fine-Tuning
Multimodal large language models need to keep learning new capabilities without forgetting old tasks—this is precisely the problem that continual instruction fine-tuning aims to solve. Mixture-of-Experts (MoE) architectures, which mitigate task interference through expert specialization and composition, appear to be a natural path forward. But existing MoE-based methods generally adopt a "one dedicated expert per task" strategy: while this avoids overwriting shared parameters, the cost is severe cross-task redundancy and linear parameter growth.
A typical example cited in the paper illustrates the problem well: both ScienceQA and TextVQA require localizing relevant regions in images, so there is clear functional overlap, yet the experts trained for each are unaware of each other and repeatedly learn similar capabilities.

The core idea of Mixture-of-Experts (MoE) architectures is to have multiple expert networks each responsible for a portion of the input space, with a router dynamically selecting or weighted-combining expert outputs based on the input. Compared with dense models, MoE activates only part of the parameters during inference, so it can control computation while increasing total parameter count. In continual learning scenarios, MoE's appeal lies in "parameter isolation": by opening new experts for new tasks, it can theoretically prevent new-task gradients from overwriting old-task parameters, thereby mitigating catastrophic forgetting. However, this naive "task-as-expert" approach ignores the functional commonality between tasks, and the number of experts grows linearly with the number of tasks, bringing redundancy and routing instability in its wake.
The Parameter Bloat Dilemma of Existing MoE Solutions
The practice of progressively expanding the expert pool is especially problematic in continual learning scenarios. The more tasks there are, the more experts are added linearly, and many of them carry overlapping functions. This is not merely a waste of storage—routing competition among redundant experts also makes expert selection unstable during inference.
A natural idea is to merge similar experts after the fact, but merging itself may destroy each task's specialized capabilities and instead induce forgetting. The design of CoPre-MoE revolves precisely around this tension: compress redundancy without touching already-learned knowledge.
Catastrophic forgetting refers to the phenomenon where, when a neural network learns a new task, the gradient updates for the new task overwrite the weights that the old task relies on, causing a sharp drop in old-task performance. Mainstream approaches to mitigating this in continual learning include regularization (constraining changes to important parameters), replay (replaying old data), and parameter isolation (allocating independent subnetworks to different tasks). MoE belongs to the parameter isolation line, but the more thorough the isolation, the greater the redundancy. Orthogonal regularization is a commonly used constraint: by penalizing the inner product between new and old parameters or representations, it pushes them toward mutual orthogonality, thereby carving out non-interfering directions in parameter space for new knowledge.
CoPre-MoE's Two-Stage Expert Compression Architecture
CoPre-MoE adopts a shared-private expert architecture, processing each newly arriving task in two steps.
Shared Expert Pool: Quantifying and Merging Functional Overlap
In the first stage, the model first learns a task-specific MoE for the current task, fully capturing its specialized capabilities. It then evaluates the degree of functional overlap between the task's experts and historical experts, and selectively merges the overlapping portions into a general expert pool with fixed capacity. The key here is "selective"—only experts whose functions genuinely overlap are merged, avoiding crude compression that damages task specialization. The fixed capacity of the general pool means the parameter scale does not keep growing with the number of tasks.
Private Expert Pool: Decomposition into Projection Basis and Compact Cores
In the second stage, the remaining task-specific experts are decomposed into a shared projection basis and compact task-specific cores, which are placed into the private expert pool. The idea behind this factorization is to extract the parts of expert weights shared across tasks and share them, retaining only the genuinely task-unique low-dimensional cores. Compared with directly storing full experts, this significantly reduces parameter count while preserving task-specific expressiveness.
Low-rank decomposition is a technique that approximates a larger weight matrix as the product of two smaller matrices, typically in the form W ≈ U·V, where the ranks of U and V are much smaller than the original matrix dimensions. In continual learning, if the expert weights of multiple tasks share the same set of "bases" (such as U) while each retains only low-dimensional "coefficients" (such as V), parameter count can be significantly compressed while preserving task-specific expressiveness. This structure of shared basis plus private core essentially separates cross-task commonality from task-unique information, and is a common means of balancing parameter efficiency and specialization capability.
Routing Design: Frozen Historical Keys + Orthogonal Regularization
The other half of the forgetting-prevention effort lies in the router. CoPre-MoE pairs each expert with a learnable retrieval key. During continual updates, the keys corresponding to historical tasks remain frozen and are not modified by new-task gradients; newly added keys are subjected to an orthogonality penalty, constraining them to be as orthogonal as possible to historical keys.
The effect of this design is that the "addressability" of historical experts in routing remains unchanged, and new tasks learn new keys in orthogonal directions, so old and new knowledge are isolated from each other in routing space. During inference, the router dynamically combines experts from the shared and private pools based on the input, and does not require task labels at test time—the model itself decides which experts to invoke.
Routing is the mechanism in MoE that determines which experts are activated for each input, typically by encoding the input with a router and computing similarity with each expert's "key" or "prototype," then obtaining weights via softmax or Top-K selection. When the expert set expands dynamically with tasks, if old experts' keys are modified by new-task training, inputs originally pointing to old experts may be misrouted to new experts, causing performance degradation. Freezing historical keys is equivalent to fixing the "addresses" of old knowledge, while the orthogonal penalty makes new keys fall in directions unrelated to historical keys; together, the two enable stable dynamic expert composition without relying on task labels.
Experimental Conclusions and Design Trade-offs
The paper concludes that CoPre-MoE achieves better performance while substantially reducing parameter overhead. From the perspective of method design, its contribution lies not in a single trick but in splitting the two usually conflicting goals of "merging redundancy" and "suppressing forgetting" into different modules: the expert pool handles compression, and the routing keys handle isolation.
What deserves attention are the trade-offs involved. The shared pool has fixed capacity, and as task scale continues to grow, whether the general expert pool can accommodate enough common capability is a question that requires empirical verification; the strength of the orthogonal regularization also affects the balance between learning new tasks and retaining old knowledge. The original text does not disclose the specific experimental setup or dataset scale, and these details await further confirmation in the full paper.
One-line Summary
CoPre-MoE balances anti-forgetting and parameter efficiency in continual instruction fine-tuning by merging overlapping experts and decomposing private experts.
Related articles

Former Huawei Genius Youth Open-Sources AI Agent Guide: 90+ Hands-On Case Studies
Former Huawei Genius Youth Li Bojie open-sources AI Agent Book, the most systematic Chinese Agent learning repo on GitHub, with 90+ runnable examples covering context compression, memory frameworks, MCP tool calling, prompt injection attack and defense, and other enterprise-level hands-on content.

The Most Detailed FDE Tutorial on Bilibili: Case-Based Teaching from Business Practice to Interviews
B站UP主推出FDE从入门到面试实战教程,讲师深入业务一线,用马氏学习平台真实案例讲解FDE岗位要求。本文分析案例教学法、笔记策略与私密信息处理,为AI岗位求职者提供参考。

518 Dance Motion Chains + H3 Relay: Long-Video Choreography Breaks the Duration Limit
The open-source dance motion library adds 518 motion chains for choreography without reference videos. The H3 workflow combines AddGuide relay, CF Lift acceleration, and VDN to enable 30–60s long-video generation, breaking single-segment duration limits with independently controlled segmented prompts.