Tensor Parallelism
Chapter 10: Matrix Block Partitioning — LLM Distrib Computing
From the book
Chapter 10: Matrix Block Partitioning. In the chapter mind map this icon labels LLM Distrib Computing: Row-wise & Column-wise Tensor Parallelism. The discussion below is excerpted and lightly edited from § Column-wise Partitioning (Tensor Parallel) in Mathematics for AI and Machine Learning.
For a weight matrix $W \in \mathbb{R}^{m \times n}$ partitioned column-wise across $k$ devices:
where $W_i \in \mathbb{R}^{m \times (n/k)}$ is stored on device $i$. The forward pass computes:
where each device computes $W_i\mathbf x$ independently, and results are concatenated. This corresponds to Perspective 1 (column-wise partitioning) of block matrix multiplication.
What this drawing shows
What you see. Shows tensor blocks split across parallel shards, representing distributed matrix or tensor computation.
In the mind map. Chapter 10 — LLM Distrib Computing: Row-wise & Column-wise Tensor Parallelism. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 10: Matrix Block Partitioning — see the mind-map node LLM Distrib Computing: Row-wise & Column-wise Tensor Parallelism.
