Neural Network Activation Functions Explained: Sigmoid, Tanh, ReLU, GELU

TL;DR: Why do neural networks need activation functions? This article explains Sigmoid, Tanh, ReLU, GELU, and Softmax in plain language, with a comparison table and selection rules.

A counterintuitive fact: you can stack 100 layers of neural network, but it has exactly the same capability as a single-layer network. Sounds like a scam? And without a little thing called an "activation function," it really would be.

If you've ever seen ReLU, Sigmoid, or Softmax flash by in a deep learning tutorial, this article is for you. No complex math, no need to be scared. By the end, you'll clearly understand what activation functions actually do, why they exist, and which one to choose—no more frantic Googling.

What Exactly Is an Activation Function?

Every artificial neuron does two things. First, it receives inputs, multiplies each input by a weight, and sums them up. Second, it passes this sum through a small function that decides how strongly the neuron should "fire." That second step is the activation function.

Imagine a bouncer at a club entrance. The neuron's sum is the noise outside the door. The bouncer (activation function) decides: too quiet, don't let them in; loud enough, come in; total chaos, still only one person at a time. Different bouncers, different rules.

Why Can't You Skip It? The Secret of 100 Layers

Multiplication and addition are called linear operations. The awkward thing about linear operations is that when you combine them... you get another linear operation. Two straight-line rules stacked together are still just one straight-line rule. So a network without activation functions, no matter how deep, can only draw straight lines through your data.

Real-world problems aren't straight lines. Recognizing cats, translating Hindi to English, predicting engine wear—all involve curves and turns. Activation functions add a bend at each layer, and these bends stack up to let the network trace almost any shape. That's the whole secret.

Layers give the network depth. Activation functions give it the ability to use that depth.

What is backpropagation? Backpropagation is the core algorithm for training neural networks, and it's also the prerequisite for understanding the vanishing gradient problem. Simply put, it works in two steps: during forward propagation, input data is computed layer by layer to produce a predicted output; then the error (loss) between the prediction and the true label is calculated. Next comes the key "backward" part—the algorithm uses the chain rule from calculus to compute, starting from the output layer and working backward, each weight's "share of responsibility" for the error, which is the gradient. Each weight is then nudged slightly in the gradient direction to make the next prediction more accurate. The problem is that the chain rule means the gradient is the product of the derivatives of each layer: if each layer's derivative is less than 1, the gradient shrinks exponentially after multiplication, which is exactly the root cause of why early layers in deep Sigmoid networks "can't learn."

Meet the Main Players

1. Sigmoid: Classic but Aging

Sigmoid compresses any number into the range 0 to 1, making it look like a probability. Large positive inputs give values close to 1, large negative inputs give values close to 0. It was the star of early neural networks because it roughly mimics a biological neuron's "fire or stay quiet."

The problem: At both ends, the curve flattens out almost completely. Flat means the slope is near zero, and the network learns by following the slope. Even at its steepest point, Sigmoid's slope is only 0.25. Multiply that number across many layers, and the learning signal shrinks to almost nothing. This is the famous vanishing gradient problem. Early layers essentially stop learning—like a game of telephone where the message is already garbled before it reaches the first person.

Where it still shines: The final layer for binary classification (spam or not spam).

2. Tanh: Sigmoid's More Balanced Sibling

Tanh (pronounced "tanch") compresses numbers into the range -1 to 1. Because the output is centered around zero, learning is usually smoother than with Sigmoid. It still flattens at both ends, so it has the same vanishing gradient headache, just slightly better. You'll still encounter it in older recurrent networks like LSTMs.

3. ReLU: The "Boring" Function That Changed Everything

ReLU stands for Rectified Linear Unit—a very grand name for a very simple rule: if the number is negative, output 0; otherwise pass it through unchanged. That's it. In code, it's max(0, x).

Why has such a humble rule come to dominate deep learning? Two reasons. It's extremely cheap to compute (just a comparison, no exponentials). And for positive inputs, the slope is exactly 1, so the learning signal doesn't decay during backpropagation. ReLU became the default after the AlexNet image model won the ImageNet competition, kicking off the modern deep learning boom.

The problem: A neuron that consistently receives negative inputs will output zero forever, and a slope of zero means it can never recover. People call this a "dead" neuron, or the dying ReLU problem. Poor thing.

The Historical Significance of AlexNet and ImageNet: In 2012, AlexNet, proposed by Alex Krizhevsky and colleagues, slashed the top-5 error rate in the ImageNet large-scale image recognition competition from about 26% the previous year to around 15%, beating the runner-up by more than 10 percentage points. This result stunned the entire computer vision community and is regarded as a landmark event in the deep learning revival. One of the key factors behind AlexNet's success was precisely the first systematic use of ReLU instead of Sigmoid/Tanh in a large-scale deep convolutional network. ReLU made training deep networks feasible, boosting training speed several times over. After that, ReLU quickly became the default choice for hidden layers, and the deep learning boom erupted in full force.

4. Leaky ReLU: Giving the Dead a Pulse

Leaky ReLU fixes the dead neuron problem with a small tweak: instead of outputting exactly 0 for negative numbers, it lets a small fraction through (usually 1% of the input). The neuron always has a slope, so it can always recover.

5. GELU and Swish: The Smooth Modernists

ReLU has a sharp corner at zero. GELU and Swish (also called SiLU) round that corner into a gentle curve, which generally trains better in very large models. GELU is used in well-known language models like BERT and the GPT series. A fun bit of trivia: Swish was discovered when researchers used automated search to try many candidate functions, rather than someone sitting down and deriving it.

Why the Transformer Architecture Favors Smooth Activation Functions: Transformer is the underlying architecture of modern large language models like BERT and GPT, with self-attention plus feed-forward networks at its core. Compared to traditional CNNs, Transformer layers are deeper and have more parameters, demanding higher gradient stability. GELU (Gaussian Error Linear Unit) smoothly "gates" inputs by introducing the cumulative distribution function of a Gaussian distribution. In the negative region, it doesn't hard-zero like ReLU but retains a small signal, making gradients more continuous. Swish achieves a similar effect by multiplying the input by a sigmoid. This smoothness significantly improves convergence speed and final performance in large-scale pretraining, making it almost the standard configuration for the feed-forward layers of Transformer-family models.

6. Softmax: The Vote Counter

Softmax is different. It doesn't process one neuron at a time; it looks at a whole group of outputs and turns them into percentages that add up to 100%. If a model has to decide between cat, dog, and bird, Softmax might output 85%, 10%, and 5%. It belongs in the final layer of a multi-class classifier, never in a middle layer.

Quick Comparison

Function	Output Range	Best For	Watch Out For
Sigmoid	0 to 1	Final layer, binary classification output	Vanishing gradient
Tanh	-1 to 1	Older recurrent networks	Vanishing gradient
ReLU	0 to infinity	Hidden layers, default choice	Dead neurons
Leaky ReLU	Slightly below 0 to infinity	When ReLU neurons die	One extra parameter to choose
GELU / Swish	Smooth, mostly positive	Transformers, large models	Slightly more computation
Softmax	0 to 1, sums to 1	Final layer, multi-class classification	Not for hidden layers

So Which One Should You Choose?

A battle-tested safe recipe that works for a surprisingly large number of projects:

1. Hidden layers: Start with ReLU. If many neurons seem dead, try Leaky ReLU.
2. Binary classification output: Sigmoid.
3. Multi-class classification output: Softmax.
4. Regression output (like predicting temperature): No activation function, just output the raw value.
5. Building a Transformer: Follow what similar models do, usually a GELU- or Swish-type function.

Try It Yourself in 30 Seconds

import numpy as np

x = np.array([-3, -1, 0, 1, 3])

sigmoid = 1 / (1 + np.exp(-x))
relu = np.maximum(0, x)
tanh = np.tanh(x)

print("sigmoid:", sigmoid.round(2))
print("tanh :", tanh.round(2))
print("relu :", relu)

Run it and compare. Notice how Sigmoid barely changes between 3 and 100, while ReLU keeps growing. That difference is exactly why one struggles with depth while the other doesn't.

FAQ

Can I use different activation functions in different layers?

Yes. Hidden layers typically use one function (like ReLU), and the output layer uses another (like Softmax) to match the task.

Is ReLU always the best?

No. It's the best starting point. Newer smooth functions may win in very large models, but for most beginner projects, ReLU is fast, simple, and hard to beat.

What is vanishing gradient in one sentence?

It's when the learning signal keeps shrinking as backpropagation passes through layers, until early layers get almost no feedback and stop improving.

Final Thoughts

An activation function is a tiny bend that turns a pile of straight-line math into something that can recognize faces, translate languages, and predict failures. Small switch, big effect. Which surprised you more: the 100-layer trick, or that ReLU is just max(0, x)? Let me know in the comments.

If you found this helpful, hit ❤️ and follow for more plain-English machine learning and engineering guides. New articles drop every Tuesday and Friday, and this one is part of the Machine Learning Fundamentals series.

This article was originally published on ML AI Insights Hub.