AdaGrad
An optimization algorithm that adapts the learning rate for each parameter individually; it is suited to sparse-gradient scenarios, but its monotonically decreasing learning rate may cause premature decay.
Timeline (last 90 days)
The video builds the optimizer step by step in the order of plain GD, Nesterov, AdaGrad, RMSProp, and Adam.
RMSProp improves on AdaGrad using an exponential moving average, avoiding premature decay of the learning rate.
All Facts (5)
AdaGrad通过累积历史梯度平方实现自适应，但会导致学习率单调递减直至停止更新
90%UnverifiedAdam可以被理解为AdaGrad和RMSProp的进化版本
80%UnverifiedRMSProp replaces accumulation with an exponential moving average, solving AdaGrad's vanishing learning rate problem.
60%UnverifiedThe video builds the optimizer step by step in the order of plain GD, Nesterov, AdaGrad, RMSProp, and Adam.
50%UnverifiedRMSProp improves on AdaGrad using an exponential moving average, avoiding premature decay of the learning rate.
50%