Chapter 15: Bellman Equations and Operators. In the chapter mind map this icon labels Markov Decision Processes: $(S, A, P, R, \gamma)$. The discussion below is excerpted and lightly edited from § Markov Decision Processes in Mathematics for AI and Machine Learning.

In the matrix chapter, we studied Markov chains where state transitions follow a fixed probabilistic law with no external control. A Markov Decision Process extends this framework by introducing an agent that can influence the process through actions. Unlike a Markov chain where transitions are purely stochastic, an MDP allows the agent to choose actions at each step, making the process controlled rather than passive. To guide the agent's decision-making, we add a reward function that evaluates the desirability of state-action pairs, and a discount factor that balances the importance of immediate versus future rewards. This transformation from a passive stochastic process to an active decision-making framework is what makes MDPs the mathematical foundation of reinforcement learning.