Full Fine-Tuning
Full fine-tuning is a fine-tuning approach that updates all parameters of a pretrained model to adapt it to a downstream task, typically incurring substantial computational and storage overhead.
Timeline (last 90 days)
Improving cross-image comparison ability does not necessarily require large-scale retraining; localizing the attention heads that carry the function and applying minimal targeted interventions may yield significant gains, offering a more economical approach than full fine-tuning for scenarios such as image-text comparison, document comparison, and visual grounding.
The 350M model can be fully fine-tuned (not just LoRA) on a GPU with only 8GB of VRAM, and for datasets and context lengths of manageable size, training takes only a few hours.
For large language models, directly updating all parameters requires storing gradients and optimizer states, and the memory overhead can reach several times the model's own parameter count.
The size of a full fine-tuning checkpoint is usually comparable to that of the base model, reaching tens of GB.
In reinforcement learning post-training, the weight changes learned by Deletion Tuning are closest to full fine-tuning on all three metrics of stable rank, spectral drift, and principal weight overlap, while LoRA and random sparse fine-tuning deviate further.
Across the three post-training paradigms of RLVR, SFT, and on-policy distillation, zeroing out less than 0.05% of the weights in the pretrained model matches the performance of full fine-tuning at scale.
The masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.
If the amount of fine-tuning data is limited, directly unfreezing all parameters for fine-tuning will almost inevitably trigger forgetting; a more reliable approach is to first freeze the backbone network and train only the final output head or adaptation layers.
The study compared three adaptation strategies: full fine-tuning, LoRA low-rank adaptation, and post-hoc weight-space merging.
All Facts (9)
Improving cross-image comparison ability does not necessarily require large-scale retraining; localizing the attention heads that carry the function and applying minimal targeted interventions may yield significant gains, offering a more economical approach than full fine-tuning for scenarios such as image-text comparison, document comparison, and visual grounding.
50%UnverifiedThe 350M model can be fully fine-tuned (not just LoRA) on a GPU with only 8GB of VRAM, and for datasets and context lengths of manageable size, training takes only a few hours.
50%UnverifiedFor large language models, directly updating all parameters requires storing gradients and optimizer states, and the memory overhead can reach several times the model's own parameter count.
50%UnverifiedThe size of a full fine-tuning checkpoint is usually comparable to that of the base model, reaching tens of GB.
50%UnverifiedIn reinforcement learning post-training, the weight changes learned by Deletion Tuning are closest to full fine-tuning on all three metrics of stable rank, spectral drift, and principal weight overlap, while LoRA and random sparse fine-tuning deviate further.
50%UnverifiedAcross the three post-training paradigms of RLVR, SFT, and on-policy distillation, zeroing out less than 0.05% of the weights in the pretrained model matches the performance of full fine-tuning at scale.
50%UnverifiedThe masks learned by Deletion Tuning are hundreds to thousands of times smaller than the corresponding full fine-tuning checkpoints.
50%UnverifiedIf the amount of fine-tuning data is limited, directly unfreezing all parameters for fine-tuning will almost inevitably trigger forgetting; a more reliable approach is to first freeze the backbone network and train only the final output head or adaptation layers.
50%UnverifiedThe study compared three adaptation strategies: full fine-tuning, LoRA low-rank adaptation, and post-hoc weight-space merging.
50%