CAGD

Conditional Anchored Generation Distillation (CAGD) is an approach targeting catastrophic forgetting in continual learning of language models; it uses the old model's predictive distribution as the distillation target, protecting model capabilities at the behavioral level rather than the data level.

CAGD decouples three roles that are conflated in data replay: the condition (selecting which behaviors to protect), teacher generation (locating where to protect), and soft targets (specifying how predictions may change).

For autoregressive language generation, CAGD's teacher unrolling distillation can precisely decompose sequence divergence via the chain rule.

The paper conducts four-task continual adaptation experiments on a masked diffusion language model with 219 million (219M) parameters.

Under one task ordering, the final held-out loss across four tasks decreased from 2.927 to 1.114; under the reverse task ordering, it decreased from 2.168 to 0.891.

When the teacher-generated support set remains exactly the same, simply replacing hard replay with soft targets can further reduce the final average loss by 0.055.

Researchers validated the approach on models such as SMDM and Qwen3 for new facts and natural instructions, and the directional conclusion that CAGD improves forgetting still holds.

On the GSM8K mathematical reasoning task, Qwen model adaptation can maintain answer format compliance, but the exact-match retention effect is confounded by random seeds at the 0.6B scale and even deteriorates at the 1.7B scale.

CAGD is more flexible than hard replay, allowing the student to produce outputs on old prompts that differ in wording from the teacher but are semantically equivalent.