Weight Decay
Weight decay is a regularization technique that adds a penalty term on the weight norm to the loss function to prevent overfitting and encourage parameters toward small values.
Timeline (last 90 days)
The study proves that for the factorized LoRA objective with weight decay, after an arbitrarily small generic positive semidefinite perturbation, every first-order stationary point is rank-deficient as long as the LoRA rank exceeds the threshold determined by the bottleneck rank.
Weight decay adds a penalty term proportional to the square of the weights to the loss function, shrinking the weights toward zero and thereby limiting the model's effective capacity.
The study reveals that when weight decay λ ≳ 1.0, the system crosses from a region where grokking does not occur to a region where it does, forming a sharp phase boundary.
Weight decay can be viewed as a knob controlling the grokking phase transition; by adjusting λ, it is theoretically possible to actively trigger or suppress grokking.
The weight decay constant λ* introduced in the work is defined as λ* = 1 − αₗᵣ·λ, where αₗᵣ is the learning rate and λ is the original weight decay coefficient.
All Facts (5)
The study proves that for the factorized LoRA objective with weight decay, after an arbitrarily small generic positive semidefinite perturbation, every first-order stationary point is rank-deficient as long as the LoRA rank exceeds the threshold determined by the bottleneck rank.
50%UnverifiedWeight decay adds a penalty term proportional to the square of the weights to the loss function, shrinking the weights toward zero and thereby limiting the model's effective capacity.
50%UnverifiedThe study reveals that when weight decay λ ≳ 1.0, the system crosses from a region where grokking does not occur to a region where it does, forming a sharp phase boundary.
50%UnverifiedWeight decay can be viewed as a knob controlling the grokking phase transition; by adjusting λ, it is theoretically possible to actively trigger or suppress grokking.
50%UnverifiedThe weight decay constant λ* introduced in the work is defined as λ* = 1 − αₗᵣ·λ, where αₗᵣ is the learning rate and λ is the original weight decay coefficient.
50%