AdamW
An improved variant of the Adam optimizer that decouples weight decay from gradient updates, avoiding the improper interaction of L2 regularization within adaptive optimizers. AdamW is a widely used optimizer in modern large-scale model training.
Timeline (last 90 days)
AdamW's effective step size is approximately proportional to η/(1-β₁), so simply increasing β₁ amounts to implicitly amplifying the effective learning rate.
In the RLVR experiments on Qwen3-4B-Base, with both the SGD+EMA and AdamW optimizers, configurations choosing a larger momentum coefficient showed an advantage.
AdamW's default value of β₁ is usually 0.9, controlling the exponential moving average window of the first-order gradient.
On the Qwen2.5-32B model, merely increasing AdamW's first-order momentum coefficient β₁ from its conventional value brought nearly a 2 percentage point improvement in average validation accuracy, without modifying other hyperparameters.
In AdamW, the parameter update is approximately equal to the first moment divided by (the square root of the second moment plus ε), and the reciprocal of the square root of the second moment acts as a preconditioner, adaptively scaling the gradient for each coordinate.
AdamW maintains two exponential moving average states, the first moment and the second moment, for each trainable parameter, usually stored in fp32, and in mixed-precision training the optimizer states often take up more GPU memory than the model weights and activations.
Compared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
The author's fine-tuning used the QLoRA method, with LoRA rank/alpha of 32/64, a learning rate of 1e-4, batch size 1 with gradient accumulation of 16, AdamW 8-bit optimizer, and training for 1 epoch.
The weight decay simplification idea in this work shares common ground with the decoupled weight decay approach in AdamW.
In the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.
1 more timeline events
All Facts (14)
DeepSeek V4采用MUON优化器替代AdamW进行训练
85%UnverifiedDecoupled DiLoCo的外部优化器通常使用Nesterov动量SGD，内部优化器通常为AdamW
85%UnverifiedMUON优化器替代了沿用多年的AdamW优化器，通过引入矩阵正交化步骤使训练曲线更加平滑
80%UnverifiedAdamW's effective step size is approximately proportional to η/(1-β₁), so simply increasing β₁ amounts to implicitly amplifying the effective learning rate.
50%UnverifiedIn the RLVR experiments on Qwen3-4B-Base, with both the SGD+EMA and AdamW optimizers, configurations choosing a larger momentum coefficient showed an advantage.
50%UnverifiedOn the Qwen2.5-32B model, merely increasing AdamW's first-order momentum coefficient β₁ from its conventional value brought nearly a 2 percentage point improvement in average validation accuracy, without modifying other hyperparameters.
50%UnverifiedAdamW's default value of β₁ is usually 0.9, controlling the exponential moving average window of the first-order gradient.
50%UnverifiedIn AdamW, the parameter update is approximately equal to the first moment divided by (the square root of the second moment plus ε), and the reciprocal of the square root of the second moment acts as a preconditioner, adaptively scaling the gradient for each coordinate.
50%UnverifiedCompared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
50%UnverifiedAdamW maintains two exponential moving average states, the first moment and the second moment, for each trainable parameter, usually stored in fp32, and in mixed-precision training the optimizer states often take up more GPU memory than the model weights and activations.
50%UnverifiedThe author's fine-tuning used the QLoRA method, with LoRA rank/alpha of 32/64, a learning rate of 1e-4, batch size 1 with gradient accumulation of 16, AdamW 8-bit optimizer, and training for 1 epoch.
50%UnverifiedThe weight decay simplification idea in this work shares common ground with the decoupled weight decay approach in AdamW.
50%UnverifiedIn the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.
50%UnverifiedThe decoder and lookup table use the AdamW optimizer (learning rate 1×10⁻⁵), while the recurrent weights use the Muon optimizer (learning rate 0.005, momentum 0.95).
50%