[KongchangAI]
Concept删除微调

Deletion Tuning

A parameter-efficient post-training method that learns binary masks on the gate projections of Transformer blocks to zero out a very small number of weights, achieving performance comparable to full fine-tuning under paradigms such as RLVR, SFT, and on-policy distillation, while reducing storage overhead and mitigating catastrophic forgetting.

Timeline (last 90 days)

Nov 10

Deletion Tuning learns a binary mask that acts on the gating projections of Transformer blocks to zero out selected weights.

Unverified50%
Nov 10

Deletion Tuning can match the performance of full fine-tuning under all three post-training paradigms: reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT) and on-policy distillation.

Unverified50%
Nov 10

Under the reinforcement learning setting of the continual learning scenario, Deletion Tuning's prior-task accuracy remains near the base model level, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 percentage points.

Unverified50%
Nov 10

On the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.

Unverified50%
Nov 10

Deploying Deletion Tuning requires saving only the base model and the binary mask, and at inference time zeroing the corresponding weights according to the mask restores the fine-tuning effect.

Unverified50%
Nov 10

Deletion Tuning is still at the research stage, and its generality across different model architectures and task types remains to be further verified.

Unverified50%
Nov 10

Deletion Tuning is a post-training method that zeroes out a very small number of weights in the attention gate projection by learning a binary mask, without adding new parameters.

Unverified50%
Nov 10

Deletion Tuning is especially suitable for scenarios that require frequent task switching or continual learning.

Unverified50%
Nov 10

The masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.

Unverified50%
Nov 10

In reinforcement learning post-training, Deletion Tuning's prior-task accuracy remains almost at the base model level, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.

Unverified50%

2 more timeline events

All Facts (12)

Unverified

Deletion Tuning learns a binary mask that acts on the gating projections of Transformer blocks to zero out selected weights.

50%
Unverified

Deletion Tuning can match the performance of full fine-tuning under all three post-training paradigms: reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT) and on-policy distillation.

50%
Unverified

Under the reinforcement learning setting of the continual learning scenario, Deletion Tuning's prior-task accuracy remains near the base model level, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 percentage points.

50%
Unverified

On the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.

50%
Unverified

Deploying Deletion Tuning requires saving only the base model and the binary mask, and at inference time zeroing the corresponding weights according to the mask restores the fine-tuning effect.

50%
Unverified

Deletion Tuning is still at the research stage, and its generality across different model architectures and task types remains to be further verified.

50%
Unverified

Deletion Tuning is a post-training method that zeroes out a very small number of weights in the attention gate projection by learning a binary mask, without adding new parameters.

50%
Unverified

Deletion Tuning is especially suitable for scenarios that require frequent task switching or continual learning.

50%
Unverified

The masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.

50%
Unverified

In reinforcement learning post-training, Deletion Tuning's prior-task accuracy remains almost at the base model level, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.

50%
Unverified

In reinforcement learning post-training, the weight changes learned by Deletion Tuning are closest to full fine-tuning on all three metrics of stable rank, spectral drift, and principal weight overlap, while LoRA and random sparse fine-tuning deviate further.

50%
Unverified

Random sparse fine-tuning performs noticeably worse than Deletion Tuning, indicating that the mask learning process plays a decisive role in performance.

50%

Source Articles