Zeroing Out Less Than 0.05% of Weights Rivals Full Fine-Tuning?

A new study proposes Deletion Tuning: zeroing out less than 0.05% of weights can rival full fine-tuning.
The article introduces a new post-training method called Deletion Tuning: by learning a binary mask on the Transformer gate projection and zeroing out less than 0.05% of the weights in a pretrained model, it can match full fine-tuning performance in RLVR, SFT, and on-policy distillation. The method adds no new parameters and does not drastically modify weight values, bringing two practical advantages: the trained mask is hundreds to thousands of times smaller than full fine-tuning checkpoints, significantly reducing storage costs for multi-task deployment; and in continual learning scenarios, forgetting is extremely low—under reinforcement learning post-training, old-task accuracy remains almost at the base model level, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points. Geometric analysis further shows that its weight changes are closest to full fine-tuning on the three metrics of stable rank, spectrum drift, and principal-weight overlap, indicating that the key lies in "which weights are deleted" rather than sparsity itself.
Does post-training of large models really require updating a huge number of parameters? A new study offers a counterintuitive answer: zeroing out an extremely small number of weights can match the performance of full fine-tuning. This method, called Deletion Tuning, learns a binary mask to "delete" some weights in the attention gate projection, not only matching full fine-tuning in performance but also greatly reducing storage overhead and catastrophic forgetting.

Core Finding: Less Than 0.05% of Weights Is Enough
The research team found that in three mainstream post-training paradigms—reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT), and on-policy distillation—zeroing out less than 0.05% of the weights in a pretrained model, with no other modifications, can match full fine-tuning (FFT) at scale.
This finding challenges the conventional wisdom that "post-training must adjust a large number of parameters." Traditional full fine-tuning updates all model parameters, while parameter-efficient fine-tuning methods (such as LoRA) reduce trainable parameters through low-rank adapters. Deletion Tuning takes a more extreme route: it adds no new parameters and does not drastically modify weight values, but instead directly "deletes" a very small number of connections.
Specifically, the method learns a binary mask on the gate projection of Transformer blocks to decide which weights are zeroed out. The mask itself is binary, and the training process can be understood as finding a set of weights whose deletion least affects performance on the current task.
Post-training paradigms refer to further training a model after pretraining, aimed at specific tasks or alignment goals. The three paradigms mentioned in the article each have their own focus: RLVR (reinforcement learning with verifiable rewards) uses automatically verifiable correct answers (such as math problems or code tests) as reward signals to guide the model in improving its reasoning ability; SFT (supervised fine-tuning) performs standard maximum likelihood training on human-annotated input-output pairs and is the most common alignment method; on-policy distillation has the student model generate samples itself, and then the teacher model provides supervision signals for these samples, balancing exploration and knowledge transfer. What all three have in common is that they require adjusting model parameters, and Deletion Tuning shows that this "adjustment" can be replaced by an extremely sparse deletion operation.
Two Practical Advantages: Storage and Resistance to Forgetting
The first direct benefit of Deletion Tuning is storage efficiency. The trained mask is hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoint. This means that when deploying multiple task-adapted versions, storage costs can be greatly reduced, and model switching becomes lighter.
The second advantage is especially critical in continual learning scenarios: Deletion Tuning suffers far less forgetting than other methods. In reinforcement learning post-training, its accuracy on prior tasks remains almost at the base model level, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points.
This result directly addresses the core challenge of continual learning—how to learn new capabilities without destroying old knowledge. Traditional full fine-tuning and LoRA both overwrite existing weights to some extent, causing performance degradation on old tasks; because Deletion Tuning deletes only a very small number of weights, its impact on the original model is confined to a very small range.
Catastrophic forgetting refers to the phenomenon in which, when a neural network learns a new task, parameter updates overwrite the weight configurations on which previous tasks depended, causing a sharp decline in performance on old tasks. This problem is especially prominent in continual learning or multi-task deployment scenarios: each time the model adapts to a new task, it may sacrifice previously accumulated capabilities. Common approaches to mitigating forgetting include regularization constraints (limiting changes to important weights), experience replay (mixing in old task data), and parameter isolation (allocating independent modules for different tasks). Deletion Tuning is close to an extreme form of parameter isolation—because only a very small number of weights are zeroed out, most of the base model's representations are fully preserved, and old task performance is naturally maintained.
Why Does "Deletion" Work? Geometric Analysis Reveals the Reason
The mask operation seems simple, but why is it enough to replace full fine-tuning? The research team approached this from the geometric properties of weight changes, analyzing them using three metrics: stable rank, spectrum drift, and principal-weight overlap.
The results show that in reinforcement learning post-training, the weight changes learned by Deletion Tuning are closest to full fine-tuning on all three metrics, while LoRA and random sparse fine-tuning deviate further. This indicates that although Deletion Tuning modifies only a very small number of weights, the direction of its impact on model representations is highly consistent with full fine-tuning—deleting those unimportant connections actually brings the model closer to the full fine-tuning solution.
This analysis also provides a new perspective for understanding sparse fine-tuning: sparsity itself is not the key; what matters is choosing which weights to delete. Random sparse fine-tuning performs noticeably worse than Deletion Tuning, showing that the mask learning process plays a decisive role.
Stable rank is a smooth alternative to matrix rank, defined as the ratio of the squared Frobenius norm to the squared spectral norm; the smaller the value, the more the matrix's energy is concentrated in a few singular value directions. Spectral drift measures the degree of change in the singular value spectrum of the weight matrix before and after fine-tuning; the smaller the drift, the gentler the "direction of perturbation" of fine-tuning. Principal-weight overlap measures the alignment between the principal singular vectors of the fine-tuned weights and those of the original weights; high overlap means the model retains its original dominant representation directions. Together, these three characterize the geometric impact of fine-tuning on model representations: although full fine-tuning changes many parameters, the direction of its changes is often highly structured, and Deletion Tuning precisely reproduces this structured direction, while low-rank adaptation or random sparsity deviates further.
Implications for Post-Training Practice
The emergence of Deletion Tuning offers a new idea for post-training: rather than adjusting a large number of parameters, it is better to precisely delete a small number of weights. This method achieves a good balance in storage efficiency, resistance to forgetting, and performance, and is especially suitable for scenarios that require frequent task switching or continual learning.
Of course, the method is still in the research stage, and its generality across different model architectures and task types remains to be further verified. But at the very least, it suggests that post-training of large models does not necessarily require "major surgery"; sometimes subtraction is more effective than addition.
One-line Summary
A new study proposes Deletion Tuning, which zeroes out less than 0.05% of weights and can match full fine-tuning.
Paragraph Summary
The article introduces a new post-training method called Deletion Tuning: by learning a binary mask on the Transformer gate projection and zeroing out less than 0.05% of the weights in a pretrained model, it can match the performance of full fine-tuning in three post-training paradigms: RLVR, SFT, and on-policy distillation. The method adds no new parameters and does not drastically modify weight values, thus bringing two practical advantages: the trained mask is hundreds to thousands of times smaller than a full fine-tuning checkpoint, significantly reducing storage costs for multi-task deployment; and in continual learning scenarios, forgetting is extremely low—under reinforcement learning post-training, old-task accuracy remains almost at the base model level, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 percentage points. Geometric analysis further shows that its weight changes are closest to full fine-tuning on the three metrics of stable rank, spectrum drift, and principal-weight overlap, indicating that the key lies in "which weights are deleted" rather than sparsity itself.
Related articles

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Pricing and Quota Breakdown
Google Gemini 4 Neon is now live, benchmark scores closely tracking Carbon, API pricing 3/15 at twice Hydrogen's rate. This article compiles Bilibili creator testing: Neon can replace 90% of Carbon scenarios, AntiGravity quota consumption is only one-third of Carbon, plus subscription channel restrictions and 2API ban risks.

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.