Discrete-time Markov Chain

Definition

A Discrete-time Markov Chain (DTMC) is a stochastic process $\{X_n,n\ge 0\}$ where the random variable at index $n+1$ depends only on the random variable at index $n$:

$P\{X_{n+1}=j\mid X_n=i,X_{n-1}=i_{n-1},\dots ,X_1=i_1,X_0=i_0\}=P_{ij}$

Interpretation

State $n+1$ depends on state $n$, and only on state $n$. The future is conditionally independent of the past given the present.

Properties

• State space can be discrete or continuous
• Time index is discrete: $n=0,1,2,\dots$
• Satisfies the Markov property

Related

• Markov Property
• Transition Probability
• Transition Probability Matrix