Spectral Drift
A metric that measures changes in the singular value spectrum of weight matrices before and after fine-tuning, used to analyze the extent to which fine-tuning affects the representational structure of a model.
Core Facts
Timeline (last 90 days)
On the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.
The study analyzes the geometric properties of weight changes using three metrics: stable rank, spectral drift, and principal weight overlap.
Over-reliance on the fine-tuned part during fine-tuning may be precisely the root cause of rare-class capabilities being obscured; directing the model's computation moderately toward the pretrained components may be a more robust fine-tuning route.
All Facts (3)
Over-reliance on the fine-tuned part during fine-tuning may be precisely the root cause of rare-class capabilities being obscured; directing the model's computation moderately toward the pretrained components may be a more robust fine-tuning route.
75%UnverifiedOn the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.
50%UnverifiedThe study analyzes the geometric properties of weight changes using three metrics: stable rank, spectral drift, and principal weight overlap.
50%