UTA
A closed-loop, adaptive learning rate decay scheduling method that decomposes the learning rate curve into shape and scale and adjusts it dynamically according to training state through continuous-time modeling; it is compatible with Adam-type optimizers and is plug-and-play.
Timeline (last 90 days)
UTA (Universal Training Adaptive) learning rate decay treats LRD as a closed-loop shape and scale problem, achieving training-state adaptation and piecewise-optimal shape through a continuous-time formulation.
UTA does not depend on the specific dynamics of SGD, can be seamlessly integrated into modern optimizers such as Adam, and is plug-and-play with negligible extra overhead.
UTA achieves consistent improvements on language models across the LLaMA architecture (from 60M to 1B parameters).
On vision tasks, UTA also improves the performance of ResNet50 and ViT-B on ImageNet.
UTA is robust to the peak learning rate, the final decay ratio, the parameter φ₀, and the fine-tuning setting, and ablation experiments confirm that it is insensitive to key hyperparameters.
The proposal of UTA marks a shift in learning rate scheduling from empirical rules to adaptive control, and is expected to reduce manual intervention and improve training efficiency.
All Facts (6)
UTA (Universal Training Adaptive) learning rate decay treats LRD as a closed-loop shape and scale problem, achieving training-state adaptation and piecewise-optimal shape through a continuous-time formulation.
50%UnverifiedUTA does not depend on the specific dynamics of SGD, can be seamlessly integrated into modern optimizers such as Adam, and is plug-and-play with negligible extra overhead.
50%UnverifiedUTA achieves consistent improvements on language models across the LLaMA architecture (from 60M to 1B parameters).
50%UnverifiedOn vision tasks, UTA also improves the performance of ResNet50 and ViT-B on ImageNet.
50%UnverifiedUTA is robust to the peak learning rate, the final decay ratio, the parameter φ₀, and the fine-tuning setting, and ablation experiments confirm that it is insensitive to key hyperparameters.
50%UnverifiedThe proposal of UTA marks a shift in learning rate scheduling from empirical rules to adaptive control, and is expected to reduce manual intervention and improve training efficiency.
50%