Expanding BERT Classification Categories: Three Incremental Fine-Tuning Strategies and Solutions to Catastrophic Forgetting

The Core Problem: Fixed Dimensions of the Classification Head

A frequently encountered practical question in the machine learning community is: if you've already fine-tuned a BERT model for 10-class classification, can you directly add 15 new categories on top of it to expand it to 25 classes? Or should you declare all 25 classes from the very beginning, even if the data for the last 15 classes won't be available until later?

This question may seem straightforward, but it cuts to the heart of a fundamental mechanism in neural network classification tasks — the Classification Head has fixed dimensions.

BERT (Bidirectional Encoder Representations from Transformers), introduced by Google in 2018, is a pre-trained language model based on the Transformer architecture. Its core consists of multiple layers of bidirectional self-attention, capable of simultaneously capturing semantic information from both left and right context around a word. For downstream classification tasks, the hidden state vector corresponding to the [CLS] token (typically 768 or 1024 dimensions) is used as the overall sentence representation, which is then passed through a linear layer (the classification head) to map to the target number of classes. The weight matrix of this linear layer has the shape [hidden_size × num_classes]. Once num_classes is fixed, the entire model's output space is locked in. When BERT performs classification, the linear layer stacked on top of the Transformer encoder has an output dimension strictly equal to the number of classes. A trained 10-class model has a final layer output dimension of 10 — it cannot directly predict classes 11 through 25 without modifying the network architecture.

Expanding categories is fundamentally a network architecture modification problem, not simply a matter of continued training. This is the technical crux of the challenge.

Comparing Three Mainstream Strategies

In practice, there are three main approaches to handle this scenario, each with its own applicable conditions and trade-offs.

Strategy 1: Declare All 25 Classes from the Start

If the full range of future categories is known in advance, the simplest approach is to design the classification head with 25 dimensions from the very beginning. Even if only 10 classes have data initially, the model is built as a 25-class classifier, with the latter 15 classes having no positive samples during initial training.

The advantage is a unified architecture with no need for future structural changes. However, the drawbacks are equally clear: categories with no data for an extended period will have near-random predictive ability and may even slightly interfere with discrimination of existing classes. If the total number of future categories is itself uncertain, this "reserved seat" approach is difficult to implement.

Strategy 2: Expand the Classification Head + Continue Fine-Tuning

A more intuitive incremental learning approach: retain the trained BERT backbone and original 10-dimensional classification head, append 15 new output neurons to the classification head to expand it to 25 dimensions. The weights corresponding to the original 10 classes are reused directly, while the 15 new neurons are randomly initialized, followed by continued fine-tuning with new data.

The critical factor here is how training data is organized. Fine-tuning with only the 15 new classes' data makes the model highly susceptible to Catastrophic Forgetting — rapidly losing its ability to recognize the original 10 classes while learning the new ones. This is the most classic persistent challenge in incremental neural network learning.

Strategy 3: Full Retraining with All Data

If compute resources and data availability permit, mixing all 25 classes of data and completely fine-tuning from the pre-trained BERT from scratch typically yields the most stable and optimal results, and is the preferred path recommended by most practitioners.

The cost is time and compute, but it completely avoids catastrophic forgetting, allowing the model to achieve balanced optimization across all classes. For scenarios with a moderate number of categories and manageable total data volume, this is usually the most reliable choice.

Catastrophic Forgetting: The Core Challenge of Incremental Learning

"Catastrophic Forgetting" is the key factor determining whether incremental fine-tuning can succeed. First systematically described by McCloskey and Cohen in 1989, it is an inherent flaw in connectionist models. The root cause is that neural networks use shared parameters to represent knowledge across all tasks: when the model updates weights via backpropagation on new data, gradient descent modifies parameters in the direction that minimizes the new task's loss — but these same parameters also carry the knowledge of old tasks. Since the optimization process has no awareness of old tasks, the weight configurations on which old tasks depend are overwritten indiscriminately. This is fundamentally different from the hippocampus-neocortex consolidation mechanism in the human brain — the brain can integrate new knowledge into long-term memory during sleep through memory replay without disrupting existing cognitive structures. To address this, both academia and engineering practice have developed the following categories of mitigation methods.

Rehearsal (Data Replay)

The most direct and effective approach: retain and mix in a portion of old class samples when fine-tuning on new classes. The main strategies for selecting old class samples are: random sampling, representative sample selection based on feature space (e.g., samples near K-means cluster centers), and generative replay (using a generative model to synthesize pseudo-samples of old classes, eliminating the need to store real data). Research shows that retaining approximately 20 samples per class can produce noticeable forgetting suppression effects. In the BERT fine-tuning context, mixing old and new data at a ratio of 1:3 to 1:5 can effectively maintain old class performance without significantly increasing training time. This is essentially a compromise between Strategy 2 and Strategy 3 — no full retraining required, but old knowledge is anchored by "reviewing the past."

Regularization Constraints (EWC)

Elastic Weight Consolidation (EWC), published in Nature by the DeepMind research team in 2017, draws its core idea from Bayesian learning. EWC approximates the posterior distribution of parameters learned from old tasks as a Gaussian distribution and uses the diagonal elements of the Fisher information matrix to measure the importance of each parameter to old tasks. During new task training, a penalty term is added to the loss function: L_new + λ·Σ_i F_i·(θ_i - θ*_i)², where F_i is the Fisher information value of the i-th parameter and θ*_i is the parameter value after training on the old task. A higher Fisher information value indicates the parameter is more critical to the old task, resulting in a stronger penalty and making the parameter harder to modify. This is equivalent to equipping each weight with a "spring" of varying stiffness, seeking a balance between plasticity and stability. These methods require no storage of old data, making them suitable for scenarios sensitive to data privacy or with storage constraints.

Parameter Isolation and Adapters (LoRA/Adapter)

With the proliferation of PEFT (Parameter-Efficient Fine-Tuning) techniques, a more elegant approach is to use Adapters or LoRA: freeze the BERT backbone and train independent lightweight adaptive modules for different sets of classes.

Adapters, proposed by Houlsby et al. in 2019, insert small bottleneck modules (typically only a few thousand parameters) after the feed-forward network in each Transformer layer, with only these inserted modules trained during fine-tuning. LoRA (Low-Rank Adaptation), proposed by Microsoft Research in 2021, takes a more elegant matrix decomposition approach: the update ΔW to the original weight matrix W is decomposed into low-rank factors, ΔW = A·B, where the rank r is much smaller than the original matrix dimensions, compressing parameter count to typically less than 1/100 of the original. The key difference between the two: Adapters insert additional modules serially, introducing inference latency; whereas LoRA's low-rank matrices can be directly merged with the original weights at inference time (W' = W + AB), incurring no additional inference overhead, making it more engineering-friendly and the mainstream approach in the era of large models. Adding new classes only requires training new adapters with virtually no impact on existing capabilities — this approach is increasingly popular in multi-task and continual learning scenarios.

Practical Recommendations: How to Choose the Right Strategy

Based on the analysis above, the following decision path can guide strategy selection:

The full range of categories is known and data will be available incrementally: prioritize declaring all 25 classes from the start, or perform full retraining once data is complete.
Compute resources are sufficient and data volume is moderate: directly fine-tune on all 25 classes' data — simple, reliable, and delivers the best results.
Incremental learning is required and old data cannot be accessed: use the expanded classification head + EWC regularization approach, and retain a small number of old samples for replay whenever possible.
Frequent category expansion is anticipated: consider a modular architecture based on LoRA/Adapter, with built-in support for continuous expansion at the design level.

One point deserves special emphasis: regardless of which strategy is chosen, evaluation must cover all classes. Many incremental fine-tuning failures occur precisely because validation is only performed on new classes, while the gradual degradation of old class performance goes unnoticed.

Conclusion

The seemingly routine requirement of "adding new categories to a classification model" actually involves a series of deep issues: network architecture modification, catastrophic forgetting, and continual learning. For models like BERT, there is no one-size-fits-all silver bullet — the optimal strategy always depends on data availability, compute budget, and the frequency of category expansion.

In most real-world projects, full retraining remains the safest choice when conditions allow; and when business requirements demand true continual learning capability, modular PEFT approaches represented by LoRA/Adapter point toward a more forward-looking direction of evolution.