Expectation-Maximization
Chapter 16: Variational Inference and Latent Variables — Expectation-Maximization
From the book
Chapter 16: Variational Inference and Latent Variables. In the chapter mind map this icon labels **Expectation Maximization. The discussion below is excerpted and lightly edited from § Expectation-Maximization** in Mathematics for AI and Machine Learning.
EM is an iterative algorithm that maximizes the log-likelihood by alternating between two steps: the E-step (Expectation) computes the exact posterior $p_{\boldsymbol\theta}(\mathbf z | \mathbf x)$ when it's tractable, and the M-step (Maximization) updates model parameters $\boldsymbol\theta$ to maximize the expected log-likelihood.
Key properties of EM:
- Uses the exact posterior in the E-step (when tractable)
- Alternates between E-step and M-step until convergence
- Guaranteed to increase the log-likelihood at each iteration
- Works well for models with conjugate priors (e.g., GMM, HMM)
- Limited to models where the posterior can be computed exactly
What this drawing shows
What you see. Log-likelihood and ELBO curves fixed; red E-step arrow then blue M-step arrow grow in sequence on the variational landscape.
In the mind map. Chapter 16 — Expectation Maximization. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 16: Variational Inference and Latent Variables — see the mind-map node Expectation Maximization.