From SGD to Adam: Using the ε Parameter for a Smooth Optimizer Transition

A Continuum View of Optimizers

In deep learning training, choosing an optimizer is often treated as a binary decision — either SGD (Stochastic Gradient Descent) or Adam (Adaptive Moment Estimation). However, a Reddit user working on an unpublished project called Symphony-S2 has put forward an insightful perspective: SGD and Adam are not two separate algorithms, but rather two extremes of the same optimization framework under different parameter values.

This work, dubbed "Jesus's Adam", centers on how to prevent optimizers from converging to odd policies early in training — a critical concern in reinforcement learning and other sensitive training scenarios.

How ε Connects SGD and Adam

A Smooth Transition from 1 to 0

The core mechanism proposed is: by gradually reducing ε (epsilon) from approximately 1 down to approximately 0, in conjunction with tuning β₂, the optimizer can smoothly transition from SGD to Adam.

The elegance of this idea lies in how it reframes ε's true role in adaptive optimizers. In standard Adam implementations, ε is typically a tiny constant (e.g., 1e-8), used solely to prevent division by zero. Here, ε is repositioned as a key dial controlling the degree of adaptivity:

• ε ≈ 1: The second-moment term in the denominator is diluted, adaptive scaling has little effect, and the optimizer behaves like pure SGD
• ε ≈ 0: The second-moment estimate fully dominates, and the optimizer reverts to standard Adam

This design means training can start from a "gentler" SGD-like footing, gradually introducing Adam's adaptive properties as training progresses.

Why This Transition Matters Early in Training

Adam has a well-known issue at the start of training: since the second-moment estimate hasn't yet stabilized, adaptive learning rates can fluctuate wildly, causing the model to prematurely converge to suboptimal, "odd" policy regions. In reinforcement learning, this kind of early misconvergence can be fatal — once a policy gets stuck in a local suboptimum, escaping it is notoriously difficult.

The author's approach is essentially a "robust-to-aggressive" progressive strategy — first establishing a good initial direction with SGD-like stable updates, then gradually unlocking Adam's acceleration. This shares philosophical ground with learning rate warmup, but operates at a deeper level: it governs not just the step size, but the fundamental behavior mode of the optimizer.

Engineering Simplifications: Bias Correction and Weight Decay

Why Bias Correction Can Be Dropped

The author notes that the bias correction terms in Adam can be omitted from both the numerator and denominator, since their impact becomes negligible after approximately 1,000 to 2,000 training steps.

This is a pragmatic observation. Standard Adam's bias correction primarily fixes the underestimation of first- and second-moment estimates early in training. Once training stabilizes, the exponential moving averages are sufficiently "warmed up," the correction factors approach 1, and retaining these terms only adds implementation complexity with no meaningful benefit.

A Simplified Parameterization for Weight Decay

The work also introduces a weight decay constant λ*, defined as:

λ* = 1 - αₗᵣ · λ

where αₗᵣ is the learning rate and λ is the original weight decay coefficient. This "parametric reduction" folds the weight decay operation into a single multiplicative constant, simplifying the per-step update expression and making the overall optimizer implementation more compact. This echoes the philosophy behind AdamW's decoupled weight decay — both aim for cleaner, more controllable regularization behavior.

Technical Assessment: Value and Limitations

The Continuous Parameterization Perspective

The most valuable contribution of this work is its continuous parameterization view of optimizers. Treating SGD and Adam as two endpoints of the same algorithmic family is not only theoretically more elegant — it also opens the door to curriculum optimization in practice: designing a scheduling curve from SGD to Adam, allowing the optimizer's characteristics to evolve dynamically throughout training.

Areas Requiring Careful Validation

It's important to emphasize that this is unpublished work, with conclusions drawn primarily from the author's practical observations on the Symphony-S2 project. Rigorous theoretical proofs and large-scale independent replication are still lacking. Several points warrant further scrutiny:

• Scheduling strategy design: How should the specific schedule for ε decaying from 1 to 0 be determined? Linear decay, exponential decay, or adaptive adjustment based on training dynamics?
• Task generalizability: Over what range of tasks does the "avoid odd policy convergence" effect hold?
• Edge case risks: Could omitting bias correction introduce problems in extreme scenarios, such as very small batch sizes or high-noise gradients?

Implications for Deep Learning Practitioners

Even setting aside the specific implementation, the core insight of this work remains valuable: an optimizer doesn't have to be a fixed black box — its internal parameters (especially oft-ignored quantities like ε) can serve as effective handles for training control. For researchers struggling with early convergence issues in reinforcement learning, experimenting with dynamic ε scheduling may well be a path worth exploring.

Summary

"Jesus's Adam" — an interesting name wrapping a pragmatic optimizer improvement idea. By smoothly interpolating between SGD and Adam through ε, complemented by dropped bias correction and simplified weight decay, it aims for more robust behavior early in training. While it remains exploratory and unpublished, its "continuous unification" perspective offers a genuinely useful lens for rethinking optimizer design. Interested readers can consult the author's publicly available Symphony-S2 project documentation on GitHub for more details.