Chapter 10: Matrix Block Partitioning. In the chapter mind map this icon labels Hessian Block Structure & GPTQ Block-wise Quantization. The discussion below is excerpted and lightly edited from § GPTQ: Block-wise Hessian for Quantization in Mathematics for AI and Machine Learning.

GPTQ (GPT Quantization) is a post-training quantization method that uses block-wise Hessian decomposition to quantize weight matrices with minimal accuracy loss. The key insight is that computing and inverting the full Hessian matrix is computationally infeasible for large weight matrices, but block-wise decomposition makes it tractable.

For a Transformer linear layer with weight matrix $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, quantization seeks to find a quantized weight matrix $\hat{W}$ that minimizes the reconstruction error:

where $H$ is the Hessian matrix of the loss with respect to the weights. For a full weight matrix, $H \in \mathbb{R}^{(d_{\text{out}} \cdot d_{\text{in}}) \times (d_{\text{out}} \cdot d_{\text{in}})}$, which is computationally intractable for large models (e.g., $d_{\text{out}} = 4096$, $d_{\text{in}} = 4096$ gives $H \in \mathbb{R}^{16,777,216 \times 16,777,216}$).

GPTQ partitions the weight matrix $W$ into blocks, typically column-wise (or row-wise), with block size $b$ (commonly $b = 32$ to $128$). For column-wise partitioning:

where each block $W_i \in \mathbb{R}^{d_{\text{out}} \times b}$ and $k = \lceil d_{\text{in}} / b \rceil$.

Instead of computing the full Hessian $H \in \mathbb{R}^{(d_{\text{out}} \cdot d_{\text{in}}) \times (d_{\text{out}} \cdot d_{\text{in}})}$, GPTQ computes block-wise Hessians: