Adam
Adam (Adaptive Moment Estimation) is an adaptive learning rate optimization algorithm for training deep learning models, proposed by Diederik Kingma and Jimmy Ba. It combines the ideas of momentum and RMSProp by simultaneously estimating the first moment (mean) and second moment (variance) of gradients to automatically adjust the learning rate for each parameter. Adam is characterized by computational efficiency, minimal hyperparameter tuning requirements, and fast convergence, and is widely applied in neural network training tasks.
Core Facts
Adam combines the momentum method with RMSprop's adaptive learning rate, independently maintaining exponential moving averages of the first-order and second-order gradients for each parameter.
The Adam optimizer was proposed in 2014, combining momentum and adaptive learning rates, and has become the most widely used optimizer today.
The model weights alone of a 175-billion-parameter model (in BF16 precision) require 350 GB of storage, and the Adam optimizer must maintain two sets of states—first-order moments and second-order moments—totaling more than 1 TB.
Timeline (last 90 days)
A nonlinear time series perspective may inspire incorporating the coupled dynamics of other adaptive optimizers (such as Adam and RMSProp) into a similar analysis.
Low-dimensional reparameterization changes the shape of gradient flow, and its interaction behavior with adaptive optimizers such as Adam requires additional validation.
Phase-HDC loses approximately 5 accuracy points on average relative to float32 Adam.
Across eleven datasets spanning images, tabular data, and text, Phase-HDC reduces memory by 16–23× compared to standard float32 Adam and by 4–6× compared to 8-bit Adam.
In experiments where all other conditions were held constant and only the update rule was replaced, Phase-HDC achieved the accuracy of 6-bit momentum Adam while reducing memory to one-third.
Low-bit Adam collapses in rare-input scenarios because the gradient signal produced by rare samples is weak, and after coarse-graining the momentum the weak signal is overwhelmed, causing the corresponding parameters to fail to update.
On six of the eleven datasets Phase-HDC achieves higher accuracy than 8-bit Adam, and on the byte-level text prediction task 8-bit Adam collapses entirely.
Adaptive optimizers such as Adam and RMSProp have partially mitigated the stiffness problem in a low-cost manner, approximating a similar effect through per-coordinate adaptive step sizes, which is simpler in engineering practice.
The study points out that Adam in practice is subject to three constraints: diagonal truncation, empirical label substitution, and temporal lag.
A study published on arXiv (arXiv:2610.00004v1) decomposes Adam's complete update rule (including the momentum term) into a diagonal empirical Fisher approximation.
22 more timeline events
All Facts (20)
Adam combines the momentum method with RMSprop's adaptive learning rate, independently maintaining exponential moving averages of the first-order and second-order gradients for each parameter.
80%VerifiedThe Adam optimizer was proposed in 2014, combining momentum and adaptive learning rates, and has become the most widely used optimizer today.
80%VerifiedThe model weights alone of a 175-billion-parameter model (in BF16 precision) require 350 GB of storage, and the Adam optimizer must maintain two sets of states—first-order moments and second-order moments—totaling more than 1 TB.
75%VerifiedGradient descent has spawned three main variants — batch gradient descent, stochastic gradient descent, and mini-batch gradient descent — and the Adam optimizer is a combined improvement built on momentum and adaptive learning rates.
65%VerifiedUsing the Adam optimizer with a default learning rate of 1e-3 as a starting point is a robust entry strategy for deep learning.
65%UnverifiedAdam的一阶矩和二阶矩的计算都基于指数移动平均（EMA）
95%UnverifiedAdam的论文标题为《Adam: A Method for Stochastic Optimization》
95%UnverifiedAdam由Diederik P. Kingma和Jimmy Ba于2014年提出，发表于ICLR 2015
95%UnverifiedAdam的二阶矩估计记录梯度平方的指数移动平均，反映梯度的波动幅度
90%UnverifiedAdam的EMA设计只需要存储上一时刻的矩估计值，内存开销极小
90%UnverifiedAdam在RMSProp基础上进一步引入一阶矩（动量），并增加了偏差修正机制
90%UnverifiedOpen Code最初叫Terminal AI Agent，由一位开发者业余时间创建，后来知名开发者Dex和Adam加入
85%UnverifiedAdam可以被理解为AdaGrad和RMSProp的进化版本
80%UnverifiedWhen using the Adam optimizer, 16 bytes of state information are required per parameter (4 bytes for FP32 parameters + 4 bytes for FP32 gradients + 4 bytes for first-order momentum + 4 bytes for second-order momentum), requiring at least approximately 24 GB of state information per GPU for a 1.5 billion parameter model.
60%UnverifiedAdamW corrects the confusion between weight decay and L2 regularization in Adam by decoupling weight decay from the gradient update, and has become the de facto standard optimizer for training Transformers.
60%UnverifiedAdaptive learning rate optimizers such as Adam and RMSProp further improve convergence speed and stability over SGD.
60%UnverifiedAdam (Adaptive Moment Estimation) is one of the most widely used optimizers in deep learning.
60%UnverifiedAdam的一阶矩衰减系数β₁通常默认为0.9
55%UnverifiedLow-dimensional reparameterization changes the shape of gradient flow, and its interaction behavior with adaptive optimizers such as Adam requires additional validation.
50%UnverifiedLow-bit Adam collapses in rare-input scenarios because the gradient signal produced by rare samples is weak, and after coarse-graining the momentum the weak signal is overwhelmed, causing the corresponding parameters to fail to update.
50%