Vanishing Gradient
A common problem in training deep neural networks. When activation functions such as Sigmoid or Tanh are used, gradients attenuate layer by layer during backpropagation, making it nearly impossible to update parameters in deeper layers. This was a primary reason early deep networks were difficult to train. The introduction of the ReLU activation function substantially mitigated this issue.
Core Facts
The vanishing gradient problem originates from saturating activation functions such as Sigmoid or Tanh, whose derivatives approach zero for large or small input values, causing gradients to decay layer by layer.
Vanishing gradient refers to the exponential decay of gradient values during backpropagation in deep neural networks, causing parameters near the input layer to receive almost no updates.
Timeline (last 90 days)
Deep networks using the Sigmoid activation function are prone to the vanishing gradient problem, making it difficult for early layers to learn.
Excessively large weights may cause the Sigmoid activation function to saturate, triggering vanishing gradients.
When the hidden dimension dk is large, the raw attention scores can spike to 20, 30 or even higher, and feeding them directly into softmax leads to output saturation and vanishing gradients.
An actionable method for detecting vanishing gradients is to monitor the ratio of weight update magnitudes across layers; if deeper layers show updates several orders of magnitude larger than shallower layers, vanishing gradients may be occurring.
Sigmoid函数的导数最大值仅为0.25，经过数十层连乘后梯度会指数级衰减
梯度消失问题由Hochreiter在1991年和Bengio等人在1994年正式分析
When the factors multiplied by the chain rule are continuously less than 1, gradients decay exponentially (vanishing); when they are continuously greater than 1, gradients grow exponentially (exploding).
All Facts (9)
The vanishing gradient problem originates from saturating activation functions such as Sigmoid or Tanh, whose derivatives approach zero for large or small input values, causing gradients to decay layer by layer.
70%VerifiedVanishing gradient refers to the exponential decay of gradient values during backpropagation in deep neural networks, causing parameters near the input layer to receive almost no updates.
70%UnverifiedWhen the factors multiplied by the chain rule are continuously less than 1, gradients decay exponentially (vanishing); when they are continuously greater than 1, gradients grow exponentially (exploding).
60%UnverifiedDeep networks using the Sigmoid activation function are prone to the vanishing gradient problem, making it difficult for early layers to learn.
50%UnverifiedExcessively large weights may cause the Sigmoid activation function to saturate, triggering vanishing gradients.
50%UnverifiedWhen the hidden dimension dk is large, the raw attention scores can spike to 20, 30 or even higher, and feeding them directly into softmax leads to output saturation and vanishing gradients.
50%UnverifiedAn actionable method for detecting vanishing gradients is to monitor the ratio of weight update magnitudes across layers; if deeper layers show updates several orders of magnitude larger than shallower layers, vanishing gradients may be occurring.
50%UnverifiedSigmoid函数的导数最大值仅为0.25，经过数十层连乘后梯度会指数级衰减
50%Unverified梯度消失问题由Hochreiter在1991年和Bengio等人在1994年正式分析
50%