From GD to Adam: An AI Engineer's Journey Through Optimizer Evolution

An AI engineer shared his first video on Reddit, attempting to build the Adam optimizer from scratch, using concrete problems to drive each step of the evolution. This post not only presents a new perspective on technical explanation but also reflects a pragmatic approach to teaching deep learning optimization algorithms.

Why Build Adam from Scratch?

According to the Reddit post, in the video the AI engineer "builds Adam from scratch, with each step driven by a specific problem from the previous step: plain GD, Nesterov, AdaGrad, RMSProp..." This narrative style differs from traditional textbooks—rather than presenting formulas first and then explaining them, it reconstructs the development of optimizers as a series of engineering problem-solving processes.

The value of this approach is that it turns Adam from a black box into something transparent. Learners can see why each component (momentum, adaptive learning rate, bias correction) was introduced and what pain points each one addresses.

The Logical Chain of Optimizer Evolution

Although the original post doesn't elaborate on details, the sequence he mentions outlines a clear logical chain:

• Plain Gradient Descent (GD): The most basic update method, but suffers from sensitivity to learning rate and slow convergence.
• Nesterov Momentum: Introduces a momentum term to accelerate convergence and reduce oscillations, while "looking ahead one step" to adjust direction.
• AdaGrad: Adapts the learning rate for each parameter, suitable for sparse gradient scenarios, but the learning rate monotonically decreases to become too small.
• RMSProp: Improves AdaGrad with exponential moving average, preventing the learning rate from decaying too early.
• Adam: Combines momentum with RMSProp's adaptive learning rate and adds bias correction, becoming one of the most commonly used optimizers today.

This chain is exactly the "problem-driven" narrative the video attempts to present: each algorithm is a response to the shortcomings of the previous one.

Bias correction is one of the key designs that distinguishes Adam from RMSProp. Adam maintains two exponential moving averages: the first moment (momentum) and the second moment (squared gradient). In the early stages of training, these two estimates are initialized to zero, causing them to be biased toward zero and thus producing a large bias. Bias correction fixes this by dividing the first and second moments by $(1-\beta_1^t)$ and $(1-\beta_2^t)$ respectively, where $t$ is the time step and $\beta_1$, $\beta_2$ are decay rates. This way, even in the first few steps, the learning rate is reasonably scaled, avoiding updates that are too small or unstable. This mechanism ensures that Adam can make rapid and effective progress even early in training, which is an important reason for its robust performance in practice.

Nesterov momentum differs from classical momentum in where the gradient is computed. Classical momentum first computes the gradient at the current parameters, then combines it with the previous momentum term; Nesterov momentum, however, first performs a "lookahead" update of the parameters based on the current momentum term, computes the gradient at the predicted next position, and then uses that gradient to correct the update direction. This "look ahead one step" strategy allows the algorithm to respond more quickly when the gradient direction changes, and theoretically provides better convergence guarantees. Intuitively, it's like taking a step in the direction of momentum first, then fine-tuning based on the gradient at the new position, thereby reducing overshoot and oscillation. This is also why Nesterov can achieve stronger convergence rates in convex optimization.

Animation and Teaching: The Power of Manim

The original post mentions that "all animations were made with Manim." Manim is a mathematical animation engine developed by 3Blue1Brown, often used for producing high-quality mathematical visualizations. For abstract concepts like optimizers that involve iterative trajectories and parameter update directions, animations can intuitively show the behavioral differences of different algorithms on the loss surface, lowering the barrier to understanding.

The author specifically requests feedback, "especially where the math is imprecise or unclear." This open attitude is commendable and reminds us that while visualization is helpful, mathematical rigor remains fundamental.

Lessons for Technical Content Creators

This engineer's attempt offers several insights for AI education content:

1. Problem-driven beats formula listing: Building knowledge from pain points is easier to digest than simply throwing out formulas.
2. Visualization is a lever for understanding: Appropriate animations can reveal the dynamic behavior of algorithms, something static text can hardly replace.
3. Building from scratch deepens understanding: Hands-on construction of an optimizer forces creators to confront every design decision, and this depth is transmitted to the audience.

Currently this video is the channel's first work, and the author hopes to receive feedback to improve future content. For readers who want to deeply understand the internal mechanisms of optimizers, this kind of from-scratch explanation is worth following.

(This article is based on information from the Reddit post; the video content was not watched. For specific mathematical derivations and animation details, please refer to the original video.)

One-line Summary

An AI engineer builds the Adam optimizer from scratch using a problem-driven approach and shares teaching insights.

Paragraph Summary

An AI engineer shared his first video on Reddit, showing how to build the Adam optimizer from scratch, with each step driven by a specific problem from the previous step, explaining the evolution from plain GD, Nesterov, AdaGrad, RMSProp to Adam. All animations in the video were made with Manim, and the author requests feedback on mathematical accuracy. The article outlines the logical chain of optimizer evolution, pointing out that this problem-driven teaching approach makes Adam no longer a black box and helps learners understand the motivation behind each component. It also summarizes three insights for technical content creation: problem-driven beats formula listing, visualization is a lever for understanding, and building from scratch deepens understanding.