LoRA+
An enhanced training method for LoRA that improves fine-tuning results by setting differentiated learning rates. Rather than applying a single learning rate to all low-rank adaptation parameters, it assigns separate rates to different parameter groups, which has been shown to accelerate convergence and improve final quality in fine-tuning of large models.
Timeline (last 90 days)
Unsloth v0.1.906 adds LoRA+ training (configured via loraplus_lr_ratio) and LoRA fine-tuning support for GPTQ models.
QLoRA reduces base model memory usage through quantization, DoRA decouples magnitude and direction, LoRA+ uses different learning rates for the two matrices, and PiSSA changes initialization via singular components.
All Facts (2)
QLoRA reduces base model memory usage through quantization, DoRA decouples magnitude and direction, LoRA+ uses different learning rates for the two matrices, and PiSSA changes initialization via singular components.
50%UnverifiedUnsloth v0.1.906 adds LoRA+ training (configured via loraplus_lr_ratio) and LoRA fine-tuning support for GPTQ models.
50%