Activation Functions in Neural Networks: Why Non-Linearity Matters
Published:
Why Activation Functions Exist
The core equation of a hidden layer is simple:
The matrix multiplication Wx + b is only an affine transformation — stacking ten of them still collapses into one, giving more parameters but no more expressive shape.
Activation functions break that collapse: they inject non-linearity, letting the network carve curved decision boundaries, represent thresholds, and model interactions a linear model cannot.
The Core Intuition
Think of a neuron as a tiny processor that first computes a score and then asks: should I pass this signal, suppress it, clip it, smooth it, or gate it?
- A step activation behaves like a binary rule.
- A sigmoid behaves like a soft probability gate.
- A tanh behaves like a centered soft gate.
- A ReLU behaves like a one-way valve: block negatives, pass positives.
That tiny local choice changes the global behavior of the whole network.
Historical Progression
How the field evolved
- Step / threshold activations: good for early perceptrons, but not differentiable enough for modern gradient-based learning.
- Sigmoid and tanh: smooth and differentiable, which made backpropagation practical, but they saturate.
- ReLU: dramatically simplified optimization and became the default for CNNs and MLPs.
- Modern smooth activations: GELU, SiLU, Swish, Mish, and gated variants improved optimization in large modern models.
Classical Families
A. Linear and Threshold Activations
These sit at the two extremes.
- Linear / Identity: does nothing; useful mainly in regression outputs.
- Step / Heaviside: flips from
0to1once a threshold is crossed.
Using the linear activation in every hidden layer forfeits the entire point of depth.
B. Squashing Functions
The first major family maps inputs into a bounded range:
- Sigmoid: maps to
[0, 1]. - Tanh: maps to
[-1, 1]and is zero-centered. - Softsign: also saturates, but more gently than tanh.
They are attractive because they are smooth and easy to differentiate. Their main weakness is saturation: for large positive or negative inputs, the derivative becomes tiny.
Concrete numerical example — sigmoid saturation:
| Input \(x\) | \(\sigma(x)\) | \(\sigma'(x) = \sigma(x)(1-\sigma(x))\) |
|---|---|---|
| 0 | 0.500 | 0.250 (maximum) |
| 2 | 0.880 | 0.105 |
| 4 | 0.982 | 0.018 |
| 6 | 0.998 | 0.002 |
| 8 | 0.9997 | 0.0002 |
Each row shows why neurons that receive large-magnitude inputs essentially stop learning — the gradient through them is nearly zero.
C. Piecewise-Linear Functions
Then came the ReLU era:
- ReLU: keeps the positive branch and zeros out the negative one.
- Leaky ReLU: small negative slope instead of a hard zero
- PReLU: learns that negative slope
- RReLU: uses a random negative slope during training
- ReLU6: same idea as ReLU, but clipped at
6 - Thresholded ReLU: stays at zero until a chosen threshold
These functions made optimization much easier because their positive branch keeps a strong gradient.
Concrete step-by-step: how ReLU saves the gradient
Imagine a single neuron receives pre-activation \(z = 1.5\) and the upstream gradient (from the loss) is \(\delta = 0.8\).
| Activation | Output | Local derivative | Gradient passed back |
|---|---|---|---|
| Sigmoid | \(\sigma(1.5) = 0.818\) | \(\sigma'(1.5) = 0.149\) | \(0.8 \times 0.149 =\) 0.119 |
| Tanh | \(\tanh(1.5) = 0.905\) | \(1 - 0.905^2 = 0.181\) | \(0.8 \times 0.181 =\) 0.145 |
| ReLU | \(\max(0,1.5) = 1.5\) | 1 | \(0.8 \times 1.0 =\) 0.800 |
ReLU passes the gradient through unchanged on the positive side. Stacked over many layers, that difference becomes enormous.
What the Shapes Are Telling You
You can often predict training behavior by looking at the curve.
| Shape pattern | What it usually implies |
|---|---|
| Flat tails | Risk of vanishing gradients |
| Hard zero region | Risk of dead neurons |
| Smooth transition | More stable optimization |
| Unbounded positive branch | Strong gradient flow for active units |
| Clipping | Better control, but less expressivity |
So activation functions are not just output transformations. They are also gradient transformations.
Gradient Perspective
The four recurring problems
| Problem | Meaning |
|---|---|
| Vanishing gradients | Derivatives become so small that early layers barely learn. |
| Exploding gradients | Derivatives become too large and make optimization unstable. |
| Dead neurons | Some ReLU units stay permanently inactive because they only see negative inputs. |
| Saturation | Sigmoid/tanh flatten for large magnitudes, so gradient flow collapses. |
Practical First Recommendations
If you are just starting, a strong first mental map is:
| Use case | Good default |
|---|---|
| Hidden layers in MLPs / CNNs | ReLU or Leaky ReLU |
| Very deep modern architectures | GELU or SiLU |
| Binary output | Sigmoid |
| Multi-class output | Softmax |
| Regression output | Linear |
Later chapters cover the smoother modern and output-layer functions in more detail.
What Can Go Wrong with Classical Activations?
Typical failure modes
| Activation | Potential problem |
|---|---|
| Step | Not useful for standard backpropagation because the derivative is zero almost everywhere. |
| Sigmoid | Saturates in the tails and causes vanishing gradients in deep hidden stacks. |
| Tanh | Zero-centered, but still saturates for large magnitudes. |
| ReLU | Can create dead neurons that never reactivate. |
| ReLU6 / clipped variants | Gain control, but can reduce expressivity if clipping is too aggressive. |
Side-by-Side Comparison: Classical Activations
Main Takeaway
Activation functions determine what signal a neuron emits and how gradients travel backward — which is why the story runs step → sigmoid/tanh → ReLU → modern smooth and gated activations.
References
- Nair, V. and Hinton, G. E. “Rectified Linear Units Improve Restricted Boltzmann Machines.” ICML 2010.
- Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016.
- Glorot, X., Bordes, A., and Bengio, Y. “Deep Sparse Rectifier Neural Networks.” AISTATS 2011.