Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Modern Activation Functions: GELU, SiLU, Mish, and Smooth Gating

8 minute read

Published:

TL;DR: Modern activations try to keep the optimization benefits of ReLU while making the transition around zero smoother and more expressive. GELU became standard in Transformers, SiLU/Swish became popular in efficient deep networks, and Mish explored even more flexible smooth non-monotonic behavior.

Why ReLU Was Not the End of the Story

Intuition First: Think of ReLU as a light switch — fully off or fully on. That simplicity is great for optimization, but sometimes you want a dimmer: something that transitions smoothly from "mostly off" to "fully on," with a meaningful response even near zero. Modern activations are dimmers — they keep ReLU's "pass positives strongly" behaviour while giving the network richer structure near the transition point.

ReLU solved a huge optimization problem, but it also introduced a blunt shape:

  • exactly zero on the negative side
  • exactly linear on the positive side
  • non-differentiable at zero

The Main Modern Idea

Instead of saying:

\[ f(x) = \begin{cases} 0, & x < 0 \\ x, & x \ge 0 \end{cases} \]

modern activations often say:

\[ f(x) \text{ should turn on smoothly and keep a useful derivative near } x = 0 \]

That makes them feel more like soft gates than hard thresholds.

Diagram comparing a hard ReLU switch with smoother GELU, SiLU, and Mish style gating
Figure 1 — ReLU flips on abruptly; GELU, SiLU, and Mish let the signal turn on gradually and keep more structure around zero.

Smooth ReLU-Like Families

ELU, SELU, and CELU

These functions keep the positive linear branch, but replace the dead negative side with a smooth saturating tail.

  • ELU: negative values bend toward a negative plateau
  • SELU: a self-normalizing variant designed to stabilize mean and variance
  • CELU: a continuously differentiable ELU-like variant

They acknowledge that “all negatives become zero” is sometimes too crude.

Key Insight — why a negative floor helps: ReLU neurons that receive consistently negative input produce zero output and zero gradient — they are effectively dead. ELU solves this by letting negative inputs produce a small but non-zero output (approaching \(-\alpha \approx -1\)). This creates a negative mean activation that pushes subsequent layers to self-correct, reducing the need for careful initialization. SELU takes this further by choosing \(\alpha\) and the scale \(\lambda\) analytically (\(\lambda \approx 1.0507\), \(\alpha \approx 1.6733\)) so that the activations' mean and variance automatically stay near (0, 1) across layers — a built-in batch-norm effect at no extra computation cost.

GELU

GELU is the activation you now see everywhere in Transformers.

\[ \operatorname{GELU}(x) \approx x \, \Phi(x) \]

where \(\Phi(x)\) is the Gaussian cumulative distribution function: instead of passing all positive signals and rejecting all negative ones, GELU keeps a value in proportion to how likely it is to be useful under a Gaussian view of the input.

Key Insight: GELU can be read as "stochastic ReLU." If neuron inputs are roughly Gaussian, then \(\Phi(x)\) is the probability that a standard normal sample is less than \(x\). So \(\operatorname{GELU}(x) = x \cdot P(\text{keep this value})\) — it applies a data-driven soft gate. At \(x=0\), exactly half the signal is gated through. At \(x=2\), roughly 97% passes. At \(x=-2\), only 3% passes. Unlike ReLU, even mildly negative values contribute a small residual signal.

Step-by-step numerical comparison — GELU vs. ReLU vs. ELU at key input values:

\(x\)ReLUELU (\(\alpha=1\))GELU\(\operatorname{GELU}'\)
−30−0.950−0.0040.020
−10−0.632−0.1590.083
00000.500
1110.8411.083
2221.9551.086
3332.9961.010

Notice how GELU preserves a small negative output near \(x=-1\) (−0.159), giving gradients a foothold even in the mildly negative region — something ReLU completely discards.

Swish and SiLU

\[ \operatorname{SiLU}(x) = x \, \sigma(x) \]

Swish is the same family idea; SiLU is the common fixed version — smooth, slightly non-monotonic, and behaving like a gated linear response.

Key Insight: \(\operatorname{SiLU} = x \cdot \sigma(x)\) has a beautiful interpretation: the sigmoid term acts as a learned data-driven gate on the identity term. When \(x\) is large and positive, \(\sigma(x) \to 1\) so SiLU behaves like identity. When \(x\) is large and negative, \(\sigma(x) \to 0\) so SiLU suppresses — but smoothly. The slight dip below zero near \(x \approx -1.28\) (SiLU minimum \(\approx -0.278\)) gives the network a small negative anchor, which empirically helps optimization.

Mish

Mish pushes the same logic further:

\[ \operatorname{Mish}(x) = x \, \tanh(\operatorname{softplus}(x)) \]

It is smooth, non-monotonic, and often visually looks like “a softer Swish with a richer negative-side bend.”

Key Insight: Mish wraps SiLU's gating idea inside a tanh, which compresses the gate values into \((-1, 1)\) before scaling by \(x\). The result is unbounded above (like ReLU/SiLU), bounded-below (minimum \(\approx -0.31\)), and has continuous higher-order derivatives. The richer curvature near zero gives optimizers more informative local slope information to work with.

Worked example — tracing a value through SiLU vs GELU vs Mish:

Let \(x = -0.5\) (a mildly negative pre-activation):

FunctionComputationOutputGradient at \(x = -0.5\)
ReLU\(\max(0, -0.5)\)00 (dead!)
GELU\(-0.5 \cdot \Phi(-0.5) \approx -0.5 \cdot 0.309\)−0.154\(\approx 0.154\)
SiLU\(-0.5 \cdot \sigma(-0.5) \approx -0.5 \cdot 0.378\)−0.189\(\approx 0.072\)
Mish\(-0.5 \cdot \tanh(\operatorname{softplus}(-0.5)) \approx -0.5 \cdot 0.393\)−0.196\(\approx 0.065\)

All three modern activations preserve a small but non-zero gradient where ReLU goes completely silent.

ELU \[ \operatorname{ELU}(x) = \begin{cases} x, & x > 0 \\ \alpha(e^x - 1), & x \le 0 \end{cases} \]
GELU \[ \operatorname{GELU}(x) \approx \frac{x}{2}\left(1 + \tanh\!\Big(\sqrt{\frac{2}{\pi}}\big(x + 0.044715x^3\big)\Big)\right) \]
Swish / SiLU \[ \operatorname{Swish}(x) = x \, \sigma(\beta x), \qquad \operatorname{SiLU}(x) = x \, \sigma(x) \]
Grid of modern activation functions including ELU, SELU, CELU, GELU, Swish, SiLU, Mish, Hard Sigmoid, Hard Tanh, Hard Swish, Bent Identity, and Arctan
Figure 2 — Modern activations mostly differ in how sharply they transition around zero and how much negative information they preserve.
−2 −1 0 1 2 1 2 0 ReLU ELU GELU SiLU Mish
ReLU, ELU, GELU, SiLU, and Mish on the same axes. Watch \(x \in [-2, 0]\): ReLU is flat at zero (dead zone), ELU saturates to a fixed floor, and GELU/SiLU/Mish preserve a smooth negative dip that carries gradient information back through the network.

Fast Approximations and Mobile-Friendly Variants

Smooth functions are more expensive than piecewise-linear ones, which is why approximation-based activations became popular in efficient models:

  • Hard Sigmoid: piecewise-linear approximation of sigmoid
  • Hard Tanh: clipped tanh-like shape
  • Hard Swish: approximation of Swish used in mobile models
−3 −2 −1 0 1 2 3 SiLU (smooth) Hard Swish (approx.)
SiLU (solid orange) vs. Hard Swish approximation (dashed blue). Hard Swish clips to zero below x=−3, uses the piecewise formula x(x+3)/6 in the middle range, and becomes linear above x=3. The two curves are nearly identical in [−1, 1] — close enough that mobile networks accept the tradeoff for faster on-device computation.

A Few More Interesting Curves

The visual grid also includes a few less standard but conceptually useful shapes:

  • Bent Identity: almost linear, but gently nonlinear near zero
  • Arctan: another smooth bounded squash
  • SELU / CELU: reminders that negative values do not have to be thrown away completely

Not default choices in modern LLMs, but reminders that activation design is about what happens around zero, in the tails, and in the derivative.

Which Ones Actually Matter Most Today?

ActivationWhy people use itMain tradeoff
GELUVery strong default in TransformersMore expensive than ReLU
SiLU / SwishSmooth, gated, stable, often great in efficient deep netsStill more expensive than ReLU
MishFlexible smooth non-monotonic responseLess standard in large production stacks
Hard SwishGood hardware-friendly approximationLess smooth than the original

For general-purpose modern MLPs, ReLU is still a valid baseline, but SiLU is worth testing.

What Can Go Wrong with Modern Activations?

ActivationPotential problem
ELU / SELU / CELUMore expensive than ReLU and more sensitive to architectural assumptions than many beginners expect.
GELUExcellent in Transformers, but often unnecessary overhead in smaller or simpler models.
SiLU / SwishSmoother optimization, but still costlier than piecewise-linear activations.
MishCan work well, but is less standardized and not always worth the extra complexity.
Hard approximationsFaster on-device, but they give up part of the smooth behavior that motivated the original function.

Common Misunderstanding

The best activation is not the one with the fanciest formula. It is the one whose shape matches:

  1. the optimization constraints,
  2. the architecture,
  3. the hardware budget,
  4. the role of that layer inside the model.
Key Insight — the convergence story: GELU, SiLU, and Mish are all smooth approximations of the same underlying idea: a data-dependent soft gate applied to the linear pre-activation. They differ mainly in how they compute the gate (Gaussian CDF, sigmoid, or tanh∘softplus) and in the precise shape of the negative dip. In practice, the differences between them are usually smaller than the difference between any of them and plain ReLU.

References

  1. Hendrycks, D. and Gimpel, K. “Gaussian Error Linear Units (GELUs).” 2016.
  2. Ramachandran, P., Zoph, B., and Le, Q. V. “Searching for Activation Functions.” 2017.
  3. Misra, D. “Mish: A Self Regularized Non-Monotonic Activation Function.” 2019.
  4. Klambauer, G. et al. “Self-Normalizing Neural Networks.” NeurIPS 2017.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed — most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually — but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first — which is faster, shorter, and free of race conditions.