Gradient Descent

An excessively large learning rate in gradient descent causes parameters to oscillate or even diverge around the optimal solution, while an excessively small rate results in extremely slow convergence.

Gradient descent iteratively adjusts weights in the opposite direction of the loss function's gradient, causing the prediction error to converge progressively.

Catastrophic forgetting is a classic problem in the field of neural networks; as early as the 1990s, researchers discovered that the performance on old tasks drops sharply when neural networks are continuously trained on new tasks

Gradient descent updates parameters through the iterative formula θ = θ - α∇J(θ), where α is the learning rate and ∇J(θ) is the vector of partial derivatives of the loss function with respect to the parameters.

Gradient descent is the most fundamental optimization algorithm in machine learning; a learning rate that is too large causes oscillation, while one that is too small results in slow convergence.

Machine learning measures the gap between predicted values and true labels through a loss function, then uses optimization algorithms such as gradient descent to gradually adjust model parameters to reduce the error.

Gradient descent has three variants: batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent.

Gradient descent is the most fundamental optimization algorithm in deep learning, and its mathematical essence is iteratively updating parameters in the direction of the negative gradient of the loss function.

The learning rate determines how large each step is: too large causes overshooting the valley bottom and divergence, while too small makes training extremely time-consuming.

Gradient descent has spawned three main variants — batch gradient descent, stochastic gradient descent, and mini-batch gradient descent — and the Adam optimizer is a combined improvement built on momentum and adaptive learning rates.

The parameter update rule for gradient descent is θ_{t+1} = θ_t - η·∇L(θ_t), where η is the learning rate, and updating parameters in the direction of the negative gradient reduces the loss function most rapidly.

Gradient descent mathematically involves partial derivatives and the chain rule, and its intuitive essence is a 'downhill' process of finding the lowest point.

Gradient descent is a local method designed for high-dimensional spaces, using first-order derivative information to descend efficiently in a huge parameter space, but it cannot guarantee a global optimum and also requires storing intermediate activations and optimizer states.

Understanding gradient descent requires three mathematical foundations: derivatives (to judge the direction of change), vectors (the position of parameters in high-dimensional space), and the effect of the learning rate on convergence speed.

Expressivity describes what a network can represent, not whether it can be learned via gradient descent.

Under large-resolution initialization, applying constant-step gradient descent on m tasks (each containing n samples) yields a frozen estimator with error Õ((m∧n)^(-1) + N^(-1))

Algorithm selection must match the nature of the problem: gradient descent is suited for smooth, differentiable, continuous optimization, while gradient-free methods are suited for discrete, step-wise, or highly noisy objective functions.

In Router Rumble, gradient descent starts from a single position and tests the four surrounding neighboring positions; because the discrete objective function causes the estimated slope to be zero, it ultimately stalls at 100 covered positions with no further improvement.

Gradient-based methods fail on non-smooth, discrete objective functions because without available gradient information there is no direction to move.

Gradient descent naturally tends to select flatter, simpler solutions from among the many solutions that perfectly fit the training set, and this implicit regularization effect is closely related to generalization ability.