Chapter 12: Optimization Methods. In the chapter mind map this icon labels Newton's Method & Hessian Inversion. The discussion below is excerpted and lightly edited from § Newton's Method & Hessian Inversion in Mathematics for AI and Machine Learning.

``{.algorithm caption="Newton's Method" label="alg:newton-method"} \Require Objective function $f$ with Hessian $H$, initial point $x^{(0)}$, step size $\alpha$ \Ensure Sequence $\{x^{(t)}\}$ converging to minimizer $x^*$ \State $t \gets 0$ \Repeat \State $x^{(t+1)} \gets x^{(t)} - \alpha H(x^{(t)})^{-1} \nabla f(x^{(t)})$ \State $t \gets t + 1$ \Until{convergence} \State \Return $x^{(t)}$ ``

the book figure compares the convergence trajectories of gradient descent and Newton's method on the same convex quadratic surface $f(x_1, x_2) = 0.5x_1^2 + 2.5x_2^2$. The red path shows gradient descent, which zig-zags due to ill-conditioning (the loss surface is elongated).

Gradient descent takes many steps because it ignores the curvature information encoded in the Hessian. The green path shows Newton's method, which uses the inverse Hessian $H^{-1}$ to account for curvature and jumps directly to the minimum in a single step (for quadratic functions).

The annotations highlight that gradient descent "zig-zags" because it ignores curvature, while Newton's method "jumps directly" by using Hessian information. This visualization makes concrete why Newton's method converges faster: it adapts the step direction and size based on the local geometry of the loss surface, effectively "preconditioning" the gradient to account for the function's curvature. However, Newton's method is computationally expensive ($O(N^3)$ for matrix inversion) compared to gradient descent ($O(N^2)$ for matrix-vector products), making it impractical for large-scale problems in deep learning.