[KongchangAI]
· 2 min read· 1,301 words

LEAF-OPD: Precise LLM Unlearning via Privileged Contexts and On-Policy Distillation

LEAF-OPD: Precise LLM Unlearning via Privileged Contexts and On-Policy Distillation

LEAF-OPD uses privileged-context on-policy distillation to teach models how to respond appropriately after unlearning, rather than simply suppressing outputs.

LEAF-OPD targets the gap in LLM unlearning where methods only suppress outputs without teaching how to respond, proposing on-policy distillation based on privileged contexts. A frozen teacher provides token-level supervision on student-generated trajectories: on forget data, the teacher carries "do not reveal target knowledge" instructions to demonstrate refusal; on retain data, no privileged context is added to preserve capabilities. The student always runs without privileged contexts, internalizing when to refuse. The paper also introduces the Supervision Signal (SS) metric, revealing that teacher supervision concentrates on the first few tokens, and reinforces it with lightweight SFT. It achieves the best Forget Quality, Model Utility, and Forget ROUGE-L on TOFU, and surpasses the runner-up by 34% in Forget Quality on RWKU.

Why Is "Unlearning" Harder Than "Deletion"?

Unlearning in large models is not simply parameter deletion. Existing methods often only suppress the model's output of target knowledge without teaching the model "how to respond when asked." This leads to an awkward situation: the knowledge appears erased, but when faced with related questions, the model may give confused, evasive, or non-compliant answers. LEAF-OPD's starting point is precisely to fill this gap—not only to make the model forget, but also to teach it how to respond appropriately when encountering knowledge that should be forgotten.

LEAF-OPD paper illustration

LEAF-OPD's Core Mechanism: Privileged Contexts and On-Policy Distillation

LEAF-OPD (LEArning to Forget from Privileged Contexts via On-Policy Distillation) adopts a "teacher-student" architecture. A frozen teacher model provides token-level supervision on responses generated by the student itself. This follows the idea of on-policy distillation: the student explores according to its own policy while the teacher corrects in real time, rather than having the student passively imitate fixed answers.

The key design lies in "Privileged Contexts." On forget data, the teacher receives additional instructions (e.g., "do not reveal knowledge about the unlearning target"), thereby guiding the student to learn refusal; on retain data, the teacher does not use privileged contexts, in order to preserve the model's original capabilities. The student always operates without privileged contexts, meaning it must internalize the judgment of "when to refuse" rather than rely on external prompts.

Supervision Signal: Measuring How Much the Teacher Actually Teaches

The paper points out a practical difficulty: the teacher often provides effective supervision only on the first few tokens of the student's trajectory, and the corrective signal for subsequent tokens decays rapidly. To address this, the authors introduce the Supervision Signal (SS) metric, which measures the strength of corrective supervision over a specified token interval. SS1 evaluates the entire trajectory, while SS0.5 evaluates the latter half, reflecting overall supervision strength and late-stage persistence, respectively. When supervision is weak, lightweight SFT can reinforce the teacher's context-based refusal behavior, significantly improving both scores.

On-Policy Distillation differs from traditional knowledge distillation primarily in the source of training data. Traditional distillation typically uses fixed samples generated by the teacher model (offline), and the student can only passively imitate; on-policy distillation instead lets the student first generate responses according to its current policy, and then the teacher provides token-by-token corrective signals on these "trajectories the student itself walked." In this way, the student's training distribution stays consistent with its inference-time distribution, avoiding the distribution shift problem where "answers seen during training cannot be generated at inference." In unlearning tasks, this property is particularly important: what the model needs to learn is to react correctly at the moment it might leak knowledge, rather than reciting the teacher's standard answer. The cost is that training requires frequent forward computations of the teacher model, resulting in relatively higher computational overhead.

Token-level supervision is the prerequisite for understanding the SS metric. Autoregressive language models generate text token by token, and during distillation the teacher can provide, at each position generated by the student, a probability distribution for "what the next token should be"—this is token-level supervision. Compared with sequence-level supervision (which only scores the whole sentence), it is finer-grained and can precisely locate where the student deviates from expectations. However, as the generated sequence grows longer, the teacher's corrective signal gradually weakens—possibly because the teacher's "approval" of the student's generated context increases, or because the corrective signal is diluted over long sequences. The SS metric precisely quantifies this decay: SS1 averages supervision strength over all token positions in the entire trajectory, while SS0.5 counts only the latter half of the trajectory. The larger the gap between them, the more supervision is concentrated at the beginning and the worse the late-stage persistence—and the more reinforcement via lightweight SFT and similar means is needed.

Benchmark Performance: Balancing Unlearning Quality and Model Utility

LEAF-OPD was evaluated on three commonly used benchmarks: TOFU, RWKU, and MUSE-News.

  • TOFU: LEAF-OPD achieves the best results on Forget Quality, Model Utility, and Forget ROUGE-L, while maintaining competitive retention performance.
  • RWKU: Forget Quality is 34% higher than the second-best method, with competitive retention quality.
  • MUSE-News: The paper mentions this benchmark for evaluation but does not provide specific numbers in the abstract.

These results show that privileged-context supervision combined with online token-level distillation can achieve a better balance between "forgetting cleanly" and "using smoothly."

TOFU, RWKU, and MUSE-News are currently the three most commonly used benchmarks in LLM unlearning research, but they emphasize different aspects. TOFU constructs question-answer pairs about fictitious authors and their works, specifically testing whether the model can "forget" information about designated authors, while using a retain set to measure whether general capabilities are damaged; metrics include Forget Quality, Model Utility, and Forget ROUGE-L (lower means less repetition of target content). RWKU is based on real-world person knowledge and evaluates unlearning robustness closer to actual deployment scenarios. MUSE-News focuses on knowledge unlearning in news corpora, examining model performance under real text distributions. Together, the three provide cross-validation of method effectiveness from three dimensions—synthetic data, real people, and real corpora—which is also why LEAF-OPD chose them as evaluation sets.

Deep Insight: Why On-Policy Distillation Suits Unlearning Tasks

Traditional unlearning methods often use gradient ascent or preference optimization to directly suppress the probability of target knowledge. The problem with such methods is that they "only destroy, not build"—the model learns not to say it, but not what it should say. LEAF-OPD's on-policy distillation lets the student receive corrections on its own generation distribution: the teacher demonstrates through privileged contexts "how to refuse when facing sensitive questions," and the student independently reproduces this behavior without privileged contexts. This is essentially behavior shaping, not mere knowledge erasure.

Another noteworthy detail is the decay of the supervision signal. The teacher provides effective supervision only at the beginning of the trajectory, meaning the latter half of long responses may deviate from expectations. The SS metric design allows researchers to quantify this phenomenon and reinforce it with lightweight SFT, providing a reusable diagnostic tool for subsequent unlearning method design.

Limitations and Outlook

The paper abstract does not disclose computational overhead, sensitivity to teacher model scale, or performance in multi-turn dialogue scenarios. Engineering details such as the specific construction of privileged contexts and the timing of SFT intervention also require consulting the full text. From the method itself, LEAF-OPD relies on a high-quality teacher model, and the teacher's capability ceiling may constrain the student's unlearning effectiveness. In addition, the SS metric is currently based on token interval partitioning, and how to adaptively determine the supervision window remains an open problem.

For practitioners concerned with AI safety and model controllability, LEAF-OPD offers a new perspective: unlearning is not deletion, but teaching the model a new response strategy.

One-line Summary

LEAF-OPD uses privileged-context on-policy distillation to teach models "how to respond appropriately after unlearning" rather than simply suppressing outputs.

Paragraph Summary

To address the gap in LLM unlearning where methods "only suppress outputs without teaching how to respond," LEAF-OPD proposes an on-policy distillation method based on privileged contexts. A frozen teacher model provides token-level supervision on student-generated trajectories: on forget data, the teacher carries instructions such as "do not reveal target knowledge" to demonstrate refusal, while on retain data no privileged context is added to preserve original capabilities; the student always runs without privileged contexts, thereby internalizing the judgment of "when to refuse." The paper also introduces the Supervision Signal (SS) metric, revealing that teacher supervision is mainly concentrated on the first few tokens of the trajectory, and reinforces it with lightweight SFT. On TOFU, it achieves the best Forget Quality, Model Utility, and Forget ROUGE-L; on RWKU, Forget Quality exceeds the second-best method by 34%, achieving a better balance between unlearning and utility.

#LLM unlearning#on-policy distillation#privileged contexts#TOFU benchmark#RWKU benchmark#knowledge forgetting
Share:

Related articles