Chapter 15: Bellman Equations and Operators. In the chapter mind map this icon labels Bellman Equations: Expectation & Optimality. The discussion below is excerpted and lightly edited from § Bellman Equations in Mathematics for AI and Machine Learning.

The Bellman equations are fundamental recursive relationships that characterize value functions in MDPs. There are two types of Bellman equations:

1. Bellman equation for state value function $V_\pi(s)$: expresses the value of a state in terms of the expected immediate reward and the discounted value of the next state. 2. Bellman equation for action value function $Q_\pi(s, a)$: expresses the value of a state-action pair in terms of the immediate reward and the discounted value of the next state-action pairs.

Both equations are fixed-point equations that can be solved iteratively, and they are mathematically equivalent—one can be derived from the other using the relationship between $V_\pi$ and $Q_\pi$.