Deletion Tuning
A parameter-efficient post-training method that learns binary masks on the gate projections of Transformer blocks to zero out a very small number of weights, achieving performance comparable to full fine-tuning under paradigms such as RLVR, SFT, and on-policy distillation, while reducing storage overhead and mitigating catastrophic forgetting.
Timeline (last 90 days)
Deletion Tuning learns a binary mask that acts on the gating projections of Transformer blocks to zero out selected weights.
Deletion Tuning can match the performance of full fine-tuning under all three post-training paradigms: reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT) and on-policy distillation.
Under the reinforcement learning setting of the continual learning scenario, Deletion Tuning's prior-task accuracy remains near the base model level, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 percentage points.
On the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.
Deploying Deletion Tuning requires saving only the base model and the binary mask, and at inference time zeroing the corresponding weights according to the mask restores the fine-tuning effect.
Deletion Tuning is still at the research stage, and its generality across different model architectures and task types remains to be further verified.
Deletion Tuning is a post-training method that zeroes out a very small number of weights in the attention gate projection by learning a binary mask, without adding new parameters.
Deletion Tuning is especially suitable for scenarios that require frequent task switching or continual learning.
The masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.
In reinforcement learning post-training, Deletion Tuning's prior-task accuracy remains almost at the base model level, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.
2 more timeline events
All Facts (12)
Deletion Tuning learns a binary mask that acts on the gating projections of Transformer blocks to zero out selected weights.
50%UnverifiedDeletion Tuning can match the performance of full fine-tuning under all three post-training paradigms: reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT) and on-policy distillation.
50%UnverifiedUnder the reinforcement learning setting of the continual learning scenario, Deletion Tuning's prior-task accuracy remains near the base model level, while full fine-tuning, LoRA and random sparse fine-tuning lose up to 5.4 percentage points.
50%UnverifiedOn the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.
50%UnverifiedDeploying Deletion Tuning requires saving only the base model and the binary mask, and at inference time zeroing the corresponding weights according to the mask restores the fine-tuning effect.
50%UnverifiedDeletion Tuning is still at the research stage, and its generality across different model architectures and task types remains to be further verified.
50%UnverifiedDeletion Tuning is a post-training method that zeroes out a very small number of weights in the attention gate projection by learning a binary mask, without adding new parameters.
50%UnverifiedDeletion Tuning is especially suitable for scenarios that require frequent task switching or continual learning.
50%UnverifiedThe masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.
50%UnverifiedIn reinforcement learning post-training, Deletion Tuning's prior-task accuracy remains almost at the base model level, whereas full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.
50%UnverifiedIn reinforcement learning post-training, the weight changes learned by Deletion Tuning are closest to full fine-tuning on all three metrics of stable rank, spectral drift, and principal weight overlap, while LoRA and random sparse fine-tuning deviate further.
50%UnverifiedRandom sparse fine-tuning performs noticeably worse than Deletion Tuning, indicating that the mask learning process plays a decisive role in performance.
50%