Unverified50% confidenceFactExact time
The paper establishes a loss decay rate for full-batch Adam on general homogeneous models, whose leading order depends only on the second-moment parameter.
1
Sources
50%
Confidence
Long-term
Relevance
11/3/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn AdamW, the parameter update is approximately equal to the first moment divided by (the square root of the second moment plus ε), and the reciprocal of the square root of the second moment acts as a preconditioner, adaptively scaling the gradient for each coordinate.71% similarUnverifiedPrior work has shown that on homogeneous models, when the learning rate schedule is of the strongly decaying form η(t) = o(t^{1/L-1}), Adam is proven to converge to the solution that maximizes the ℓ∞ margin.71% similarUnverifiedAdam's bias correction is performed by dividing the first and second moments by (1-β1^t) and (1-β2^t) respectively, where t is the time step and β1 and β2 are the decay rates.70% similarUnverifiedGradient descent computes the partial derivatives of the loss function with respect to model parameters and updates the parameters in the direction opposite to the gradient to approach the minimum of the loss function.70% similarUnverifiedThe coefficient α of the load balancing loss is usually taken to be around 0.01, but the specific value is highly dependent on model scale and data distribution.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1129597API
curl https://kongchang.com/api/v1/knowledge/claims/1129597MCP
get_claim(id=1129597)