ReLU

The derivative of the ReLU activation function is always 1 or 0, and together with the gradient shortcut pathway provided by residual connections in ResNet, these are the two key innovations for addressing vanishing gradients.

The mathematical definition of the ReLU activation function is f(x) = max(0, x), which can effectively mitigate the vanishing gradient problem compared to Sigmoid and Tanh.

ReLU, defined as max(0, x), rapidly gained widespread adoption in the 2010s due to its computational efficiency and gradient stability, becoming the default activation function in modern deep learning.

The vanishing gradient problem can be addressed by replacing activation functions (e.g., substituting ReLU for sigmoid), using residual connections, or adjusting the network architecture.

Kaiming initialization (He initialization) sets the variance proportional to twice the inverse of the input dimension, designed specifically for ReLU and its variants.

Recommendations for choosing hidden layer activation functions: start with ReLU, and if a large number of neurons die, try Leaky ReLU; use Sigmoid for binary classification output; use Softmax for multi-class output; use no activation function for regression output.

ReLU has a slope of 1 in the positive input region, and the learning signal does not decay during backpropagation.

A ReLU neuron that continuously receives negative input will permanently output 0 and cannot recover, i.e. the 'dead' neuron problem.

AlexNet was the first to systematically use ReLU instead of Sigmoid/Tanh in a large-scale deep convolutional network, making training deep networks feasible and increasing training speed several times over.

He initialization targets ReLU and its variants, scaling the variance to twice that of Xavier.

Common solutions for vanishing gradients include replacing sigmoid with ReLU and its variants, batch normalization, and residual connections (the core idea behind ResNet).

Attempts to replace Softmax with ReLU (ReLUformer) can train, but require additional normalization tricks to stabilize.

When the activation function is ReLU, the target function is a pure spherical harmonic, and under Gaussian universality assumptions, the student's error scales approximately as the square of the teacher's error

Spline activation functions require table lookup or iterative interpolation at inference time, and their computational cost can be several to dozens of times that of ReLU, requiring careful evaluation in latency-sensitive scenarios such as real-time speech recognition.

The core of the self-built card game AI system is a fully connected action-value network with the architecture 1857→128→64→32→16→1, using ReLU for hidden layers and a linear output layer, with approximately 248,705 trainable parameters.

A fully connected network using the ReLU activation function essentially constructs a piecewise linear function.

ReLU is defined as f(x) = max(0, x), meaning the output is zero when the input is negative and the input is passed through unchanged when it is positive.

The maximum number of segments of a single-layer ReLU network equals 1 plus the width of that layer.

The maximum number of segments in a multi-layer network is the product of the number of segments of each layer; for example, a network with architecture 3 3 has a maximum of 4×4=16 segments.

Compared to Sigmoid or Tanh activation functions, ReLU has a constant gradient of 1 on the positive half-axis, effectively alleviating the vanishing gradient problem in deep network training, while also having extremely low computational cost.