Chapter 1: Vector Space and Inner Product. In the chapter mind map this icon labels Dot Product & Projection. The discussion below is excerpted and lightly edited from § Dot Product and Similarity in Mathematics for AI and Machine Learning. Related material also appears in Chapter 12 (Algos: Constrained Optim), Chapter 8 (Least Squares & Energy Conservation).

The most common inner product is the dot product (also called scalar product) in $\mathbb{R}^d$:

The dot product is a specific instance of a more general concept. We now define inner products formally.

In $\mathbb{R}^d$ with the standard dot product, we have the following geometric relationship:

where $\theta$ is the angle between the two vectors, and $\|\cdot\|_2$ denotes the $\ell_2$ (Euclidean) norm, which will be defined in Section the referenced section. This relationship shows that the dot product measures both the magnitudes and the alignment (angle) between vectors. When vectors are normalized (unit length), the dot product reduces to $\cos\theta$, which is known as the cosine similarity:

Cosine similarity is a fundamental measure of similarity in machine learning, information retrieval, and natural language processing. Both the dot product and cosine similarity are used for measuring similarity in different contexts: the dot product is sensitive to both direction and magnitude, making it suitable when vector magnitudes carry meaningful information (e.g., Transformer attention mechanisms use dot products by default), while cosine similarity focuses purely on directional alignment and is preferred when magnitudes should be normalized away (e.g., in text similarity tasks where document length should not affect similarity scores).

Coordinate independence: The dot product is a geometric property that does not depend on the choice of coordinate system. While the component-wise formula $\langle \mathbf{u}, \mathbf{v} \rangle = \sum_{i=0}^{d-1} u_i v_i$ appears to depend on coordinates, the geometric interpretation $\langle \mathbf{u}, \mathbf{v} \rangle = \|\mathbf{u}\|_2 \cdot \|\mathbf{v}\|_2 \cos\theta$ shows that it depends only on the magnitudes and the angle between vectors—quantities that are invariant under coordinate transformations. The same two vectors will have the same dot product regardless of which orthonormal basis is used to represent them (orthonormal bases are formally defined in @eq:orthonormal-basis), as long as the dot product is computed with respect to that basis.