Activation Derivative
Chapter 11: Matrix Calculus — Activation Functions and the Derivatives
From the book
Chapter 11: Matrix Calculus. In the chapter mind map this icon labels Activation Derivs: Sigmoid, Tanh, Softmax. The discussion below is excerpted and lightly edited from § Activation Functions and the Derivatives in Mathematics for AI and Machine Learning.
The following table summarizes commonly used activation functions in neural networks and their derivatives, which are essential for backpropagation and gradient-based optimization.
| Activation | Formula | Derivative |
|---|---|---|
| Sigmoid | $\sigma(z) = \frac{1}{1 + e^{-z}}$ | $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ |
| Tanh | $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2\sigma(2 \cdot z) - 1$ | $\tanh'(z) = 1 - \tanh^2(z) = \text{sech}^2(z)$ |
| ReLU | $\text{ReLU}(z) = \max(0, z)$ | $\text{ReLU}'(z) = 1$ if $z > 0$, $0$ if $z \leq 0$ |
| Leaky ReLU | $\text{LeakyReLU}(z) = \max(\alpha z, z)$ | $\text{LeakyReLU}'(z) = 1$ if $z > 0$, $\alpha$ if $z \leq 0$ |
| ELU | $\text{ELU}(z) = z$ if $z > 0$, $\alpha(e^z - 1)$ if $z \leq 0$ | $\text{ELU}'(z) = 1$ if $z > 0$, $\alpha e^z$ if $z \leq 0$ |
| GELU | $\text{GELU}(z) = z \cdot \Phi(z)$ | $\text{GELU}'(z) = \Phi(z) + z \cdot \phi(z)$ |
| Swish | $\text{Swish}(z) = z \cdot \sigma(z)$ | $\text{Swish}'(z) = \sigma(z) + z \cdot \sigma(z)(1 - \sigma(z))$ |
| Mish | $\text{Mish}(z) = z \cdot \tanh(\text{Softplus}(z)) = z \cdot \tanh(\ln(1 + e^z))$ | $\text{Mish}'(z) = \frac{e^z (4(z+1) + 4e^{2z} + e^{3z} + e^z(4z+6))}{(1 + e^z)^2 (1 + e^{2z})}$ |
| GEGLU | $\text{GEGLU}(z) = z \odot \text{GELU}(z)$ | $\text{GEGLU}'(z) = \text{GELU}(z) + z \cdot \text{GELU}'(z) = \text{GELU}(z) + z(\Phi(z) + z \cdot \phi(z))$ |
| ReGLU | $\text{ReGLU}(z) = z \odot \text{ReLU}(z)$ | $\text{ReGLU}'(z) = 2z$ if $z > 0$, $0$ if $z \leq 0$ |
| SwiGLU | $\text{SwiGLU}(z) = z \odot \text{Swish}(z) = z^2 \cdot \sigma(z)$ | $\text{SwiGLU}'(z) = 2z \cdot \sigma(z) + z^2 \cdot \sigma(z)(1 - \sigma(z))$ |
| Softplus | $\text{Softplus}(z) = \ln(1 + e^z)$ | $\text{Softplus}'(z) = \sigma(z) = \frac{1}{1 + e^{-z}}$ |
| Softmax | $\text{Softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}$ | $\nabla_{\mathbf{z}} \text{Softmax}(\mathbf{z})_{ij} = \text{Softmax}(\mathbf{z})_i(1 - \text{Softmax}(\mathbf{z})_i)$ if $i = j$, $-\text{Softmax}(\mathbf{z})_i \cdot \text{Softmax}(\mathbf{z})_j$ if $i \neq j$ |
Notation notes:
- Variable $z$ vs $x$: In activation functions, $z$ typically denotes the pre-activation value (the input to the activation function), which is usually the output of a linear layer: $z = \mathbf{w}^\top \mathbf{x} + b$ or $z = W\mathbf{x} + \mathbf{b}$. The symbol $x$ is typically reserved for the raw input data. This convention distinguishes between the original input ($x$) and the transformed value ($z$) that enters the activation function.
- GELU symbols: For the Gaussian Error Linear Unit (GELU), $\Phi(z)$ denotes the cumulative distribution function (CDF) of the standard normal distribution: $\Phi(z) = \int_{-\infty}^z \phi(t) dt = \frac{1}{2}\left(1 + \text{erf}\left(\frac{z}{\sqrt{2}}\right)\right)$, and $\phi(z)$ denotes the probability density function (PDF) of the standard normal distribution: $\phi(z) = \frac{1}{\sqrt{2\pi}}e^{-z^2/2}$. Here, $\text{erf}(z) = \frac{2}{\sqrt{\pi}}\int_0^z e^{-t^2} dt$ is the error function (also called the Gauss error function), which is a special function related to the normal distribution. The error function satisfies $\text{erf}(-z) = -\text{erf}(z)$ and $\lim_{z \to \infty} \text{erf}(z) = 1$.
- Element-wise product: In the formulas above, $\odot$ denotes the element-wise product (Hadamard product). For vectors $\mathbf{z}, \mathbf{w} \in \mathbb{R}^{N \times 1}$, $[\mathbf{z} \odot \mathbf{w}]_i = z_i w_i$ (see the matrix chapter for details). When applied element-wise to a vector, $\mathbf{z} \odot f(\mathbf{z})$ means each element $z_i$ is multiplied by the corresponding element $f(\mathbf{z})_i$.
What this drawing shows
What you see. Shows the local slope of a nonlinear activation, used when explaining backpropagation and chain-rule gradients.
In the mind map. Chapter 11 — Activation Derivs: Sigmoid, Tanh, Softmax. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 11: Matrix Calculus — see the mind-map node Activation Derivs: Sigmoid, Tanh, Softmax.