MAEM: Breaking Recency Bias in Graph Learning with Replay-Free Dynamic Mixture of Experts

The Core Pain Point of Continual Learning in Graph Neural Networks: Recency Bias

Graph Neural Networks (GNNs) excel at handling graph-structured data, but in class-incremental graph learning scenarios, when new classes arrive successively, the model suffers from catastrophic forgetting. More critically, predictions tend to be dominated by the most recently learned classes, producing severe recency bias.

Traditional approaches face a dilemma. Dynamically expanding architectures can isolate knowledge across sessions, but most methods assume a task-incremental setting, requiring task identity (task ID) at test time to select branches or heads — which is simply unavailable in class-incremental learning. Learnable routers seem like an alternative, but continuously fine-tuning the router on new data itself biases routing toward the most recent components.

Class-incremental graph learning is a specific setting of continual learning on graph data: the model receives newly emerging node classes session by session, and at test time must classify across all seen classes without being given task identity (it doesn't know which session a sample belongs to), so the model must determine on its own which part of its knowledge to use for answering. This contrasts with the more relaxed task-incremental learning, where the task ID is known at test time and the corresponding branch or head can be directly selected, greatly reducing difficulty. Another related concept is catastrophic forgetting: when a neural network performs gradient updates on a new task, it substantially rewrites weights associated with old tasks, causing a sharp drop in old-task performance. In graph data, due to topological dependencies between nodes, representations of old classes are further perturbed by message passing from new classes, making forgetting often more severe than in the image domain.

MAEM's Three Core Designs

To address the above dilemma, researchers proposed MAEM — a replay-free, capacity-controllable dynamic mixture-of-experts framework for class-incremental graph learning. It tackles the problem simultaneously from three levels: representation learning, routing mechanism, and capacity management.

SGC Representation + Frozen Experts: Isolating Knowledge to Prevent Forgetting

MAEM uses Simplified Graph Convolution (SGC) to construct node representations, which is lighter and more stable than standard GCN, making it suitable for continual learning scenarios. Each time a new session arrives, MAEM trains a dedicated expert to handle the current classes, and immediately freezes that expert upon completion of training.

The significance of the freezing strategy is: new knowledge can only enter new experts, and the parameters of old experts are not shaken. This architecturally severs the forgetting pathway caused by gradient backpropagation, protecting prior knowledge more rigidly than regularization methods.

Simplified Graph Convolution (SGC) is an extreme simplification of standard GCN: it removes the nonlinear activations between layers, reducing multi-layer propagation to power multiplication of the normalized adjacency matrix — that is, K-order neighborhood aggregation followed by a linear classifier. The benefits are threefold: first, parameter count and computation are greatly reduced, suitable for repeated invocation in continual learning; second, after removing nonlinearity, propagation is equivalent to a fixed linear smoothing operator, making representations more stable and less prone to drastic drift as new data arrives; third, its linear nature facilitates theoretical analysis and feature reuse. In class-incremental scenarios, stable representations mean that node embeddings of old classes won't be repeatedly distorted by new sessions, complementing the "frozen expert" strategy: SGC ensures the feature space itself changes little, while freezing ensures old expert parameters aren't overwritten.

MAE Routing: Drop Task IDs, Let Reconstruction Error Speak

Routing is the most critical component of dynamic mixture of experts. MAEM doesn't use a shared trained router — because that's precisely the source of routing-level recency bias — instead, it equips each expert with a Masked Autoencoder (MAE).

Test nodes are assigned to the expert with the lowest reconstruction error. Intuitively, an expert reconstructs more accurately on the class distributions it saw during training; new-class nodes are naturally drawn to the just-trained expert, while old-class nodes continue to match old experts. This decentralized routing mechanism requires no task ID and no shared parameters, fundamentally suppressing routing-level recency bias.

Masked Autoencoder (MAE) is a class of self-supervised reconstruction models: during training, a portion of the input (such as graph node features or subgraph structure) is randomly masked, and the model is required to reconstruct the masked content from the remaining visible parts. Its core assumption is that a model capable of accurately reconstructing a certain data distribution must have internalized the patterns of that distribution. In MAEM, each expert is equipped with an MAE that takes nodes and their neighborhood features as input, using reconstruction error to measure how "familiar" that expert is with the current node. Selecting the expert with the lowest error during routing is equivalent to asking "which expert understands this node best." Since each expert's MAE is trained independently with unshared parameters, the addition of new experts doesn't change old experts' reconstruction ability, thereby avoiding the routing-level recency bias where a shared router gets pulled off course by new data. Note that MAE inference requires one forward pass per expert, so the overhead increases linearly with the number of experts.

Merging and Distillation After Capacity Is Reached: Fusing Knowledge Without Storing Old Data

The expert pool cannot grow indefinitely, otherwise model capacity spirals out of control. When a fixed capacity limit is reached, MAEM does two things:

1. Merge the least-used experts: Use Git Re-Basin alignment to align the parameter spaces of multiple low-frequency experts before merging, avoiding performance loss from simple averaging.
2. Distill into a fused expert: Use topology-aware pseudo-data synthesized from MAE distributions (DeepInversion style) to distill the merged experts into a unified fused expert. The entire process stores and revisits no historical graph data.

This means MAEM's capacity is always bounded, and it operates under a strict replay-free constraint — which is especially important for privacy-sensitive or storage-constrained real-world scenarios.

Git Re-Basin is a model parameter alignment and merging technique, originating from research on permutation symmetry in neural network loss landscapes: different networks with the same function may have different neuron orderings, and direct element-wise averaging destroys functionality. Git Re-Basin finds permutation matrices in weight space to align multiple models' neuron orderings to the same "base," then performs linear interpolation or averaging, preserving each model's capabilities as much as possible during merging. DeepInversion is a data-free distillation method: fix a trained model and optimize random noise inputs so that their activation statistics in the model (such as BatchNorm statistics) approximate those of real data, synthesizing pseudo-samples usable for distillation. MAEM borrows this idea, first synthesizing topology-aware pseudo-graph data using distributions learned by MAE, then having the fused expert learn the outputs of merged experts on this pseudo-data, achieving knowledge transfer without touching the original historical graph data.

Experimental Results: Systematic Advantages on Four Benchmarks

MAEM was validated on four class-incremental graph learning benchmarks, with results showing:

• Recency bias is significantly alleviated, with prediction distributions no longer dominated by recent classes;
• Performance surpasses multiple strong continual graph learning baselines;
• Model capacity remains bounded under strict replay-free conditions.

It's worth noting that MAEM doesn't rely on task identity, doesn't store historical data, and doesn't share a trainable router — these three "don'ts" correspond exactly to the three most common constraints when deploying class-incremental graph learning. This makes its methodological significance extend beyond mere accuracy improvement.

Methodological Insights: Decentralized Routing + Capacity Budget Are Transferable Ideas

What leaves the deepest impression about MAEM is not any single-point innovation, but how it decomposes "forgetting prevention" across three architectural levels: the representation level uses SGC to ensure stability, the routing level uses independent MAEs to eliminate the bias source, and the capacity level uses merging and distillation to achieve budget control. This layered defense approach holds reference value for continual learning in other modalities as well.

Of course, the extra inference overhead from MAE routing and the computational cost of Git Re-Basin alignment when the number of experts is large remain engineering issues to consider in practical deployment. The paper abstract doesn't disclose specific capacity thresholds and merge trigger frequencies; the sensitivity of these hyperparameters awaits further experimental validation.

One-line Summary

MAEM uses frozen experts, independent MAE routing, and merging-and-distillation to alleviate recency bias in class-incremental graph learning without replay.

Paragraph Summary

This article introduces MAEM, a dynamic mixture-of-experts framework for class-incremental graph learning. To address recency bias where the model is dominated by the most recent classes as new classes arrive, MAEM tackles the problem from three angles: at the representation level, it uses lightweight and stable SGC to build node embeddings, and trains a dedicated expert for each session before immediately freezing it, architecturally severing the forgetting pathway; at the routing level, it equips each expert with an independent MAE, selecting the one with the lowest reconstruction error as the prediction expert, requiring no task ID and avoiding a shared router being pulled off course by new data; at the capacity level, when the expert pool reaches its limit, it uses Git Re-Basin alignment to merge low-frequency experts, then distills them into a fused expert using pseudo-data synthesized from MAE distributions, all without storing historical graph data. On four benchmarks, MAEM significantly alleviates recency bias, surpasses multiple strong baselines, and maintains bounded capacity.