Continuous-Time Markov Chain
Chapter 15: Bellman Equations and Operators — Continuous-Time Markov Chains
From the book
Chapter 15: Bellman Equations and Operators. In the chapter mind map this icon labels Continuous-Time Markov Chains. The discussion below is excerpted and lightly edited from § Continuous-Time Markov Chains in Mathematics for AI and Machine Learning.
The discrete-time Markov chains we have studied so far make transitions at fixed time steps $t = 0, 1, 2, \ldots$. Continuous-time Markov chains (CTMCs) extend this framework to allow transitions at any time $t \in [0, \infty)$. This generalization is mathematically natural and connects Markov chains to differential equations.
What this drawing shows
What you see. States $S_1, S_2$ and rate $\lambda\,dt$ fixed; a token travels the forward arc then the reverse arc, illustrating CTMC jumps in continuous time.
In the mind map. Chapter 15 — Continuous-Time Markov Chains. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 15: Bellman Equations and Operators — see the mind-map node Continuous-Time Markov Chains.