Chapter 3: Subspaces and Orthogonality. In the chapter mind map this icon labels RoPE. The discussion below is excerpted and lightly edited from § Rotary Position Embedding (RoPE) in Transformers in Mathematics for AI and Machine Learning. Related material also appears in Chapter 6 (RoPE & Augmentation).

Rotary Position Embedding (ROPE)[^rope] encodes positional information in Transformers using complex number rotations. For a query vector $\mathbf{q}_m \in \mathbb{R}^{d}$ at position $m$ and a key vector $\mathbf{k}_n \in \mathbb{R}^{d}$ at position $n$, ROPE splits each $d$-dimensional vector into $d/2$ pairs: $(\mathbf{q}_m^{(0)}, \mathbf{q}_m^{(1)}), (\mathbf{q}_m^{(2)}, \mathbf{q}_m^{(3)}), \ldots, (\mathbf{q}_m^{(d-2)}, \mathbf{q}_m^{(d-1)})$. Each pair $(q_m^{(2i)}, q_m^{(2i+1)})$ is treated as a complex number $q_m^{(2i)} + i \cdot q_m^{(2i+1)}$ and rotated by angle $m \theta_i$:

where $\theta_i = 10000^{-2i/d}$ is the base frequency for dimension pair $i$. The same rotation is applied to the corresponding key vector pairs with angle $n \theta_i$.

The rotation matrix $\begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$ corresponds to multiplication by $e^{i\theta} = \cos\theta + i\sin\theta$ in the complex plane, connecting to the eigenvalues of orthonormal matrices: rotations have eigenvalues $e^{\pm i\theta}$ with $|e^{\pm i\theta}| = 1$. When computing the attention score $(\mathbf{q}_m^\text{rot})^\top \mathbf{k}_n^\text{rot}$, the rotation angles combine such that the result depends on the relative position difference $m-n$ rather than absolute positions. This relative position encoding emerges naturally from the geometric properties of rotations, enabling the model to capture positional relationships without explicit position embeddings.

ROPE's reliance on complex rotations provides theoretical grounding: the rotation preserves vector norms (since $|e^{i\theta}| = 1$), and the relative position encoding generalizes to sequences longer than those seen during training, as the rotation angles scale continuously with position.

[^rope]: Su et al., [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864), 2021. \url{https://arxiv.org/abs/2104.09864}