Chapter 11: Matrix Calculus. In the chapter mind map this icon labels Fréchet Derivative. The discussion below is excerpted and lightly edited from § Fréchet Derivative in Mathematics for AI and Machine Learning.

For matrix-valued functions $f : \mathbb{R}^{M \times N} \to \mathbb{R}^{P \times Q}$, we have seen that the Jacobian can be represented as a $(PQ) \times (MN)$ matrix via vectorization. However, there is an alternative and often more elegant viewpoint: treating the derivative as a linear operator acting directly on matrices, without vectorization.

The Fréchet derivative (also simply called the derivative) of a matrix-valued function $f$ at a point $X$ is a linear operator $f'(X)$ that maps matrices to matrices. It satisfies the first-order approximation:

where $\Delta \in \mathbb{R}^{M \times N}$ is a perturbation matrix, and $o(\|\Delta\|)$ denotes terms that vanish faster than $\|\Delta\|$ as $\|\Delta\| \to 0$. The notation $f'(X)[\Delta]$ means applying the linear operator $f'(X)$ to the matrix $\Delta$. When the context is clear (i.e., there is only one matrix variable), we use $\Delta$ without a subscript. When multiple matrices are involved and clarity is needed, we use subscripts such as $\Delta_X$, $\Delta_A$, etc., to indicate which matrix is being perturbed. This notation is consistent with the perturbation notation used in the matrix chapter for rank-1 perturbation analysis (e.g., backward error analysis where $A + \Delta_A = QR$)[^frechet-notation].

Relationship to Vectorization: The Fréchet derivative and the vectorized Jacobian are equivalent representations. The vectorized Jacobian $\nabla_X f(X) \in \mathbb{R}^{(PQ) \times (MN)}$ is the matrix representation of the linear operator $f'(X)$ with respect to the standard basis after vectorization. However, the operator viewpoint often provides more insight into the structure of derivatives, especially for functions like matrix inverses, matrix exponentials, and other matrix operations.