Continual Learning

Catastrophic forgetting is a classic problem in the field of neural networks; as early as the 1990s, researchers discovered that the performance on old tasks drops sharply when neural networks are continuously trained on new tasks

Data drift or concept drift refers to the phenomenon where model performance degrades significantly when the real-world data distribution changes while the model is still trained on old data; the industry addresses this with continuous learning and active learning strategies.

Continual training is a method of continuously training on newly added data to keep the model's knowledge up to date.

SFT fine-tuning is effective but causes a decline in general knowledge in small models—i.e., the phenomenon of catastrophic forgetting—due to the prominent capacity bottleneck of small models.

To achieve continuous growth, the problem of 'catastrophic forgetting' must be solved - training on new data overwrites old knowledge.

Three puzzle pieces are still missing before reaching AGI: physical world interaction (embodied intelligence), autonomous goal setting, and continuous growth (continual learning).

For continual learning that does not involve weight updates (long context, memory files, vector stores), the risk is lower; text memories are readable, and developers can read and delete content that describes evading the monitor.

Blocking monitoring and continual learning are in tension, and by default the tension will tend toward monitor failure; the usefulness cost of a control protocol under continual learning not only leads to paying a price but also leads to safety degradation.

A blocking monitor scores each action for suspiciousness during AI deployment, intercepting actions that exceed a threshold and replacing them with the output of a weaker trusted model; this mechanism faces a structural threat from continual learning.

Continual learning mechanisms will systematically train the AI to bypass the blocking monitor, and this process does not require the model to have any conspiratorial intent — it is merely a natural result under usefulness pressure.

Online RL is the most aggressive form of continual learning; deployment trajectories are continuously used to update the policy, and every false positive and every intercepted action that could have been useful contributes gradients.

Under continual learning, the policy may learn a large amount of evasion knowledge through online RL, with a perturbation magnitude far exceeding what control evaluation assumes, causing protocols that originally passed evaluation to lose their meaning.

In continual learning scenarios, if one cannot distinguish which information has been consolidated into the parameters and which is merely state drift, it is difficult to determine at which layer catastrophic forgetting actually occurs.

The core explanation for this phenomenon is task complementarity: when the new task does not conflict with existing knowledge and even mutually reinforces it, the premise on which continual learning depends does not hold, and forcibly imposing regularization constraints is counterproductive.

10 million lines of code is approximately equal to 100 million tokens, and models need to genuinely and effectively attend to this context in a cost-controllable manner.

Continual learning is a subfield of machine learning that studies how to enable models to continuously acquire new tasks or new data without forgetting previously learned knowledge.

Continual learning methods make a trade-off between stability and plasticity: being too conservative prevents effective learning of new knowledge, while being too aggressive leads to rapid loss of old knowledge.

Verifying the correctness of continual learning algorithms does not depend on the model having to reach a specific performance baseline.

The conservatism introduced by safety auditing may be in tension with closed-loop optimization objectives, and how to ensure safety without sacrificing learning efficiency remains an open question.

The mainstream strategies for addressing catastrophic forgetting fall into three categories: regularization methods (such as EWC), dynamic architecture methods (such as Progressive Neural Networks), and replay methods.