Chapter 6: Geometric Transformation and Eigen Decomposition. In the chapter mind map this icon labels Nonlinear Activations. The discussion below is excerpted and lightly edited from § Nonlinear Transformations in Mathematics for AI and Machine Learning.

Not all transformations are linear. A nonlinear transformation does not satisfy the linearity properties above. Crucially, nonlinear transformations cannot be represented by a matrix—this is exactly what distinguishes them from linear transformations. While every linear transformation $T: \mathbb{R}^N \to \mathbb{R}^M$ can be written as $T(\mathbf{x}) = A\mathbf{x}$ for some matrix $A$, nonlinear transformations require more general functions.

• Polynomial transformations: $T(\mathbf{x}) = [x_0^2, x_1^2, \ldots]^\top$
• Exponential transformations: $T(\mathbf{x}) = [e^{x_0}, e^{x_1}, \ldots]^\top$
• Trigonometric transformations: $T(\mathbf{x}) = [\sin(x_0), \cos(x_1), \ldots]^\top$
• Softmax function: Applied row-wise to a matrix, $\text{softmax}(A)_{ij} = \frac{e^{a_{ij}}}{\sum_k e^{a_{ik}}}$

the book figure illustrates three examples of nonlinear transformations applied to a regular grid:

1. Original grid (left): A uniform $11 \times 11$ grid in the plane. 2. Non-equidistant transformation (middle): A polynomial-like transformation where $x' = x^2/10$ and $y' = y$. This compresses the grid near $x=0$ and stretches it near $x=10$, demonstrating how nonlinear transformations can create non-uniform spacing that cannot be achieved with linear transformations. 3. Twisted transformation (right): A trigonometric transformation where $x' = x + \sin(y/2)$ and $y' = y + \sin(x/2)$. This creates curved, wavy grid lines, showing how nonlinear transformations can introduce curvature and distortion.

Unlike linear transformations, which preserve straight lines and uniform spacing (up to scaling and rotation), nonlinear transformations can bend, compress, and distort the grid in complex ways. This flexibility is essential in machine learning, where nonlinear activation functions enable neural networks to learn complex, non-linear decision boundaries.