How to Merge Two Neural Networks? Model Merging and Catastrophic Forgetting Explained

The Problem: When Two Specialized Models Want to Become One

A Reddit user raised a question that seems simple on the surface but cuts to the heart of a core deep learning challenge: they have two neural networks with identical architectures — Network A specializes in recognizing cats, outputting a confidence score for whether an image contains a cat; Network B specializes in recognizing dogs. Both have the same input/output dimensions, and each has been compressed down to the minimum number of layers needed to complete its task. The goal: merge these two networks into one without increasing the model size, while avoiding catastrophic interference and catastrophic forgetting.

This constraint is extremely demanding. To understand why it's so hard to satisfy, we first need to understand where neural networks store knowledge, and what "merging" actually means mathematically.

Why "In-Place Fusion" Is Nearly Impossible

A neural network's knowledge is distributed across all of its weight parameters. A single weight in Network A may simultaneously contribute to the extraction of multiple features — it can't be isolated and spliced like a database field. When two networks each use their full capacity to fit their respective tasks, they've essentially "filled up" the parameter space.

Insufficient Capacity Is the Root Contradiction

The question's author emphasizes that both networks use the "absolute minimum number of layers needed for their specific task." This is precisely the crux of the problem. If Network A's capacity is just enough to recognize cats, and Network B's capacity is just enough to recognize dogs, then a network of the same size theoretically cannot hold both tasks simultaneously.

Catastrophic forgetting is a direct product of this capacity competition: when you try to learn a new task on an already-trained network, gradient updates overwrite the weights that the old task depended on, effectively "washing away" old knowledge. Under the minimum-capacity assumption, this overwriting is nearly unavoidable.

Why Simply Averaging Weights Fails

A naive approach is to average the weights at corresponding positions in the two networks (weight averaging). This sometimes works under specific conditions — for example, when two models are fine-tuned from the same initialization — but for networks that are independently and randomly initialized and trained from scratch, averaged weights typically produce a poorly-performing hybrid. The reason is that the neuron arrangements and internal representations of the two networks are completely different; direct averaging is like forcibly superimposing two incompatible coordinate systems.

Viable Technical Approaches

Although the hard constraint of "no size increase" is difficult to satisfy perfectly, there are several established methods in the field that can achieve model merging with acceptable trade-offs.

Option 1: Expand Slightly and Train Jointly (Most Reliable)

The most straightforward approach is to relax the "exact same size" constraint, build a slightly larger network with shared lower-level feature extraction layers, and add two output heads (multi-head) at the top. Low-level convolutions and feature extraction tend to be general-purpose for both cats and dogs (edges, textures, shapes) — the high-level layers are what differentiate the tasks. You then train jointly on both cat and dog data. This approach essentially eliminates catastrophic forgetting because both tasks are optimized simultaneously.

Option 2: Knowledge Distillation

Treat Networks A and B as two "teachers" and train a new "student" network to simultaneously mimic both outputs. The student aligns with A's output on cat images and with B's output on dog images. The advantage of distillation is that it can compress the knowledge of two models into one, and the student network's size can be set freely. If you insist on minimum capacity, performance after distillation may degrade somewhat — but that's a reasonable trade-off under capacity constraints.

Option 3: Neuron Alignment for Model Merging

To address the failure of naive weight averaging, researchers have proposed permutation alignment methods such as Git Re-Basin. The core idea is: before averaging weights, find a neuron permutation that aligns the internal representations of the two networks as closely as possible, then interpolate or average. These methods have achieved significantly better results than naive averaging for merging isomorphic networks, and represent an important advance in the model merging field in recent years.

Git Re-Basin (proposed by Ainsworth et al. in 2022) is a landmark work in permutation alignment. Its key insight is that neural networks have "permutation symmetry" — reordering the neurons in a given layer, as long as you synchronously adjust the weights in adjacent layers, leaves the network's function completely unchanged. So two networks with the same architecture but trained independently each occupy different "symmetric copies" in high-dimensional weight space that are functionally equivalent. Git Re-Basin uses the Hungarian algorithm or stochastic gradient descent to find a neuron permutation matrix that makes Network B's activation distribution as close as possible to Network A's, effectively "rotating" B into A's coordinate system before interpolating. Experiments show that after alignment, the loss landscape between the two networks often exhibits a linear mode, and the interpolated model can maintain reasonable performance — whereas naive averaging typically falls into high-loss "barrier regions." However, this method is less effective on heterogeneous tasks with large distributional differences (such as the cat vs. dog scenario with different classes) compared to same-task fine-tuning scenarios, so alignment quality should be verified in practice.

Option 4: Parameter-Efficient Task Vectors

The Task Arithmetic / Task Vector approach, popular in the era of large models, is also worth considering: treat the weight changes introduced by fine-tuning as a "task vector," and combine multi-task capabilities through vector addition and subtraction. However, these methods typically assume the models share a common pre-training starting point, which limits their applicability to two small networks trained independently.

Task Arithmetic (Ilharco et al., 2023) subtracts the pre-trained weights from the fine-tuned model weights to obtain a "task vector" that encodes task-specific knowledge. Multiple task vectors can be added together and applied on top of the pre-trained weights, enabling a single model to perform multiple tasks with almost no additional training. The effectiveness of this method rests on a key premise: multiple fine-tuned models share the same pre-training starting point, so their weight differences are all expressed in the same semantic coordinate system, making vector addition meaningful. For the scenario described — two small networks trained independently from random initialization — there is no common "origin," so task vector methods cannot be directly applied. But if the scenario changes to: first pre-train a backbone on large-scale general data, then separately fine-tune for cat recognition and dog recognition, Task Arithmetic becomes an extremely lightweight merging tool that can merge the two in one step without any retraining.

Practical Advice for the Original Poster

If the "exactly the same size" constraint is truly non-negotiable, the honest answer is: under the minimum-capacity assumption, lossless merging is mathematically nearly infeasible, because you're asking a fixed-capacity system to carry twice the information.

A more practical path is to re-examine the constraints:

• If a slight size increase is acceptable, shared backbone + multi-output head joint training is the most mature and effective engineering solution;
• If small size must be maintained and reacquiring training data isn't possible, try weight merging after neuron alignment, but expect some performance degradation;
• If sufficient data is available, knowledge distillation can preserve both capabilities as much as possible within a controlled model size.

Catastrophic forgetting is not unsolvable — it is the core long-term research topic in the deep learning field of continual learning. Related methods such as Elastic Weight Consolidation (EWC) and experience replay can serve as directions for further exploration. But no method can circumvent the information-theoretic capacity constraint — this is the hard physical limit that the "minimum size" assumption in the original question is bound to face.

Background: Elastic Weight Consolidation (EWC)

Elastic Weight Consolidation (EWC) is one of the classic methods for mitigating catastrophic forgetting, proposed by Kirkpatrick et al. in 2017 at DeepMind. The approach draws on Bayesian inference: it uses the Fisher Information Matrix to estimate how important each weight is to the old task, then applies an elastic penalty to important weights when learning a new task, preventing them from drifting too far from their old values. Intuitively, it's like "tightening a spring around the important screws." EWC's limitation is that as the number of tasks grows, the constraint terms keep accumulating, increasing computational and memory overhead. More critically, when capacity itself is insufficient to accommodate two tasks, EWC can only slow the rate of forgetting — it cannot fundamentally solve the capacity bottleneck. This is precisely the unavoidable limitation under the "minimum size" assumption in the scenario described.