SPIN Optimizer: A Stability-Plasticity Interpolation Scheme Tailored for Online Continual Learning

Title

SPIN Optimizer: A Stability-Plasticity Interpolation Scheme Tailored for Online Continual Learning

Meta Description

This article introduces SPIN, a novel optimizer designed specifically for Online Continual Learning (OCL). By interpolating plasticity and stability sequences, it implicitly achieves inverse Hessian approximation and Tikhonov regularization, significantly improving the performance of multiple OCL methods under rehearsal-free and strict memory constraints.

Article Content

Online Continual Learning (OCL) requires models to learn from non-stationary data streams under strict single-pass constraints, making them highly susceptible to catastrophic forgetting. Although prior research has explored various strategies from data, architecture, and optimization perspectives, optimizers often default to general-purpose methods such as SGD and AdamW. A recent paper proposes a novel optimizer, SPIN (Stability-Plasticity INterpolation), designed specifically for OCL. By explicitly decoupling plasticity and stability, it achieves significant performance improvements across multiple OCL methods.

Why Are General-Purpose Optimizers Insufficient for OCL?

The core challenge of OCL is that the model must continuously learn new knowledge from an ever-changing data stream while not forgetting old knowledge. This "stability-plasticity" dilemma makes general-purpose optimizers difficult to apply directly: SGD and AdamW are not designed for single-pass, non-stationary data distributions, and their update rules cannot effectively balance rapid adaptation with anti-forgetting requirements. The paper points out a new connection between optimization dynamics and OCL requirements—treating Polyak-Ruppert averaging as the engine of "plasticity" and Primal averaging as the anchor of "stability." This perspective provides a theoretical foundation for designing specialized optimizers.

Polyak-Ruppert averaging is a classic parameter averaging technique: during optimization, instead of directly using the parameters from the last step, it takes the arithmetic mean of historical iterates (or the latter half of them). Theory shows that in stochastic gradient descent, this averaging reduces the variance caused by gradient noise, leading to more stable convergence, and is often regarded as a form of implicit regularization. In the OCL context, the paper maps this to the "plasticity" engine—because it preserves the model's ability to rapidly adapt to recent data while smoothing the update trajectory. Primal averaging, on the other hand, refers to directly averaging the original parameter sequence rather than the dual or gradient space. It emphasizes the accumulation and anchoring of historical states, and is thus used by the paper as the "stability" anchor to help the model resist forgetting in single-pass data streams. The interpolation between the two constitutes the core mechanism by which SPIN balances stability and plasticity.

SPIN's Design: Interpolating Two Sequences

The core idea of SPIN is to explicitly decouple plasticity and stability by interpolating between a fast-moving "plasticity" sequence and an adaptive "stability" sequence. Specifically, it maintains two parameter sequences: one updated rapidly to capture patterns in new data (plasticity), and the other providing a stable anchor through an averaging mechanism (stability). The optimizer interpolates between the two, thereby dynamically balancing new and old knowledge during single-pass learning. This design allows SPIN to serve as a "plug-and-play" component for existing OCL methods—simply feed the gradients produced by the method as input and replace the standard optimizer.

Theoretical Support: Implicit Inverse Hessian Approximation and Tikhonov Regularization

The paper theoretically proves that SPIN implicitly employs an inverse Hessian approximation, which provides a key Tikhonov regularization effect. In OCL, since each sample appears only once, Hessian estimates are typically computed from a single sample and are extremely noisy. SPIN's implicit regularization can dampen these noisy estimates, thereby mitigating forgetting. This theoretical analysis not only explains SPIN's effectiveness but also connects it to classical second-order optimization and regularization techniques, providing an extensible framework for future research.

Tikhonov regularization is a classic method for solving ill-posed problems. By adding a parameter norm penalty (such as L2 regularization) to the objective function, it constrains the solution space, making the solution more stable and unique. In second-order optimization, it is equivalent to adding a positive definite term (such as λI) to the Hessian matrix, thereby improving the condition number and suppressing noise amplification. In OCL, single-sample Hessian estimates are extremely noisy, and direct inversion leads to drastic fluctuations in parameter updates; the inverse Hessian approximation implicitly introduced by SPIN acts similarly to Tikhonov regularization, damping these noisy directions and making updates smoother, thereby mitigating catastrophic forgetting. This connection links SPIN to classical methods such as ridge regression and the Levenberg-Marquardt algorithm, providing a theoretical explanation for its effectiveness.

Experimental Results: Consistent Gains Across Methods and Settings

Extensive experiments show that replacing the standard optimizer with SPIN yields consistent and significant performance improvements across multiple OCL methods. Notably, these gains hold even under challenging rehearsal-free settings and strict memory constraints. This means SPIN is not only applicable to methods with replay buffers but can also be effective in scenarios where storing old samples is impossible, which is particularly important for privacy-sensitive or storage-constrained real-world applications.

Implications for OCL Research and Practice

The proposal of SPIN reminds us that the design of the optimizer itself may have been long underestimated in continual learning. Past research has mostly focused on data replay, regularization, or architectural isolation, whereas SPIN demonstrates that by re-examining optimization dynamics, performance gains can be achieved without additional memory or computational complexity. For practitioners, SPIN's plug-and-play nature lowers the barrier to adoption; for researchers, it brings optimizer design back to the core agenda of OCL, potentially inspiring more specialized optimization algorithms for non-stationary data streams.

Limitations and Future Directions

Although the paper demonstrates SPIN's broad effectiveness, its generalization ability across different modalities (such as vision and language) and more complex task streams still requires further validation. In addition, the interaction mechanisms when SPIN is combined with replay-based methods, hyperparameter sensitivity, and computational overhead on large-scale models are all worth exploring. Future work could focus on these directions to further advance the development of OCL optimizers.

One-line Summary

SPIN optimizer explicitly decouples the OCL dilemma and improves performance by interpolating plasticity and stability sequences.

Paragraph Summary

This article introduces SPIN, a novel optimizer designed specifically for Online Continual Learning (OCL). To address the difficulty general-purpose optimizers (such as SGD and AdamW) face in balancing stability and plasticity, SPIN maintains a rapidly updated plasticity sequence and an averaging-based stability sequence, achieving explicit decoupling by interpolating between them. Theoretical analysis shows that SPIN implicitly employs an inverse Hessian approximation, providing a Tikhonov regularization effect to dampen single-sample Hessian noise. Experiments demonstrate that SPIN can replace standard optimizers as a plug-and-play component, achieving consistent and significant performance improvements across multiple OCL methods, rehearsal-free settings, and strict memory constraints. This work brings optimizer design back to the core agenda of OCL and discusses future directions such as generalization and hyperparameter sensitivity.