EWC

The three main strategies for mitigating catastrophic forgetting are regularization-based methods (such as EWC), replay-based methods (such as Experience Replay), and architecture-based methods (such as Progressive Neural Networks).

Elastic Weight Consolidation (EWC) estimates the importance of parameters to old tasks via the Fisher information matrix and applies stronger regularization penalties to important parameters to mitigate catastrophic forgetting.

To mitigate catastrophic forgetting, practices often use methods such as replay buffers to retain historical samples, elastic weight consolidation to constrain the magnitude of updates to important parameters, or parameter-efficient fine-tuning (such as LoRA).

The retention mechanism implementation path of RSI-Anything is closer to system-level component validation than to parameter-level regularization constraints (such as EWC or PNN)

The mathematical form of EWC is to add a penalty term on top of the new task loss, taking the form L_new + λΣF_i(θ_i - θ_i*)^2, where θ* is a snapshot of the old task parameters, λ controls the degree of protection, and F_i comes from the Fisher information matrix.

Using EWC in a task-complementary scenario to lock parameters near the old optimal position hinders the model from fully absorbing financial-domain-specific patterns, thereby lowering overall performance.

Standard EWC typically uses a diagonal Fisher information matrix to approximate parameter importance, which has low computational cost but assumes parameter independence, ignores inter-parameter correlations, and is too coarse an approximation for highly coupled neural networks.

In scenarios where regularization is needed, mini-KFAC EWC can achieve lower word error rates and a more stable optimization process compared to standard diagonal EWC.

The experiments in this article are simplified 2D toy experiments using extremely small-scale networks and only two tasks; task conflicts and weight importance estimation are more complex in real large-scale models.

In the EWC experiment, without using any data from Task A, Task A accuracy dips midway to 97% before recovering and stabilizing at 100%, while Task B reaches 100%, achieving no persistent forgetting.

EWC was proposed by Kirkpatrick et al. in 2017; its theoretical foundation comes from Bayesian inference, using the Fisher information matrix to estimate the importance of each weight to old tasks.

The regularization term of EWC takes the form λ/2 × Σ Fᵢ(θᵢ - θ*ᵢ)², where θ*ᵢ is the weight value after training on the old task, Fᵢ is the Fisher information, and λ controls the constraint strength.

When capacity is insufficient to accommodate two tasks, EWC can only slow down forgetting without fundamentally resolving the capacity bottleneck, and the overhead increases as the number of tasks grows due to the accumulation of constraint terms.

The mainstream approaches to addressing catastrophic forgetting fall into three categories: parameter isolation methods (such as Progressive Neural Networks), regularization-based methods (such as the EWC elastic weight consolidation algorithm), and experience replay methods.

The mainstream strategies for addressing catastrophic forgetting fall into three categories: regularization methods (such as EWC), dynamic architecture methods (such as Progressive Neural Networks), and replay methods.