Bellman Backup
From the book
Chapter 15: Bellman Equations and Operators. In the chapter mind map this icon labels Policies: Stochastic $\pi(a|s)$ & Value $V(s)$. The discussion below is excerpted and lightly edited from § Definition: State Value Function in Mathematics for AI and Machine Learning.
The state value function $V_\pi: \mathcal{S} \to \mathbb{R}$ under policy $\pi$ is the expected return starting from state $s$:
where the expectation is over trajectories generated by following policy $\pi$.
What this drawing shows
What you see. Root state $s$ fixed; successor backups from $s_1$ and $s_2$ grow upward, illustrating the Bellman backup tree.
In the mind map. Chapter 15 — Policies: Stochastic & Value. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 15: Bellman Equations and Operators — see the mind-map node Policies: Stochastic & Value.
