Chapter 11: Matrix Calculus. In the chapter mind map this icon labels Gradient: $\nabla f(\mathbf{x}) = [\frac{\partial f}{\partial x_0}, \dots, \frac{\partial f}{\partial x_{N-1}}]^\top$. The discussion below is excerpted and lightly edited from § Definition: Gradient in Mathematics for AI and Machine Learning. Related material also appears in Chapter 12 (Gradient Descent & Convergence Rate).

The symbol $\nabla$ (pronounced "nabla") is a vector differential operator. For a function $f : \mathbb{R}^N \to \mathbb{R}$, the operator $\nabla_{\mathbf{x}}$ measures the rate of change (slope) of $f$ in each coordinate direction separately, and then synthesizes these partial derivatives into a single vector. This resulting vector is the gradient.

Formally, for $\mathbf{x} \in \mathbb{R}^N$ and $f : \mathbb{R}^N \to \mathbb{R}$, the gradient is defined as:

For matrix-valued inputs, if $X \in \mathbb{R}^{M \times N}$ and $f : \mathbb{R}^{M \times N} \to \mathbb{R}$, the gradient is:

Both $\nabla_{\mathbf{x}} f(\mathbf{x})$ and $\nabla_{\mathbf{x}} f$ are commonly used notations for the gradient. The form $\nabla_{\mathbf{x}} f(\mathbf{x})$ explicitly indicates that the gradient is evaluated at the point $\mathbf{x}$, while $\nabla_{\mathbf{x}} f$ emphasizes that the gradient is an operator acting on the function $f$. Both notations are equivalent and acceptable; the choice is often a matter of style or emphasis.