Unverified50% confidenceFactExact time
Llama 2 used the AdamW optimizer (beta1 = 0.9, beta2 = 0.95, weight decay 0.1, gradient clipping 1.0), and was pretrained on 2 trillion tokens.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/17/2026
First Seen
Valid until: 1/15/2027
Sources
Related Entities
Related Claims
UnverifiedAdamW's effective step size is approximately proportional to η/(1-β₁), so simply increasing β₁ amounts to implicitly amplifying the effective learning rate.72% similarUnverifiedAdam 优化器由 Kingma 和 Ba 于 2014 年提出，同时维护梯度的一阶矩估计与二阶矩估计，并通过偏差修正项处理训练初期矩估计偏向零的问题70% similarUnverifiedUsing the program's own gradients and the Adam optimizer, a Handwired twin network was trained on 2000 random orders; when only prices were changed, by step 100 it had exactly learned 549, 949, and 350 per kilogram, with zero error.69% similarUnverifiedThe trained model achieves a per-token negative log-likelihood (NLL) of 0.93, compared to 12.60 for a randomly initialized twin model of the same architecture.69% similarUnverifiedAdamW's default value of β₁ is usually 0.9, controlling the exponential moving average window of the first-order gradient.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1049788API
curl https://kongchang.com/api/v1/knowledge/claims/1049788MCP
get_claim(id=1049788)