Mean-Field Approximation
Chapter 16: Variational Inference and Latent Variables — Mean-Field Variational Inference
From the book
Chapter 16: Variational Inference and Latent Variables. In the chapter mind map this icon labels Mean-Field Variational Inference. The discussion below is excerpted and lightly edited from § Mean-Field Variational Inference in Mathematics for AI and Machine Learning.
We have seen three choices of variational family; the simplest is mean field. Here we assume the latent variables are independent, so $q(\mathbf z)$ factorizes into a product of univariate marginals. That factorization buys tractability: we can optimize each factor separately, often in closed form. In this section we proceed step by step:
1. State the assumption—make the mean field factorization precise. 2. Derive the ELBO under this factorization, so we see how the objective simplifies. 3. Present coordinate-ascent variational inference (CAVI)—an algorithm that iteratively updates each $q_i(z_i)$ while holding the others fixed, with updates that are often analytic.
Mean field works well when correlations between latents are weak; when they are strong, the independence assumption can miss important structure. We will see a geometric illustration of this limitation in the figure below.
What this drawing shows
What you see. Correlated posterior $p$ (blue ellipse) fixed; dashed factorized $q$ expands, illustrating $q(z)=\prod_i q_i(z_i)$ mean-field VI.
In the mind map. Chapter 16 — Mean-Field Variational Inference. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 16: Variational Inference and Latent Variables — see the mind-map node Mean-Field Variational Inference.