[KongchangAI]
Conceptspectrum drift

Spectral Drift

A metric that measures changes in the singular value spectrum of weight matrices before and after fine-tuning, used to analyze the extent to which fine-tuning affects the representational structure of a model.

Core Facts

Timeline (last 90 days)

Nov 10

On the three geometric metrics of stable rank, spectral drift and principal weight overlap, Deletion Tuning is closer to full fine-tuning than LoRA and random sparse fine-tuning.

Unverified50%
Nov 10

The study analyzes the geometric properties of weight changes using three metrics: stable rank, spectral drift, and principal weight overlap.

Unverified50%
Oct 16

Over-reliance on the fine-tuned part during fine-tuning may be precisely the root cause of rare-class capabilities being obscured; directing the model's computation moderately toward the pretrained components may be a more robust fine-tuning route.

Verified75%

All Facts (3)

Source Articles