Continuous-Time Markov Chain

Definition

Let $\{X(t):t\ge 0\}$ be a continuous-time stochastic process taking values in the set of nonnegative integers. The process is a Continuous-Time Markov Chain (CTMC) if for all $s,t\ge 0$ and nonnegative integers $i,j,x(u)$ with $0\le u<s$:

$$\mathrm{Pr}\{X(t+s)=j\mid X(s)=i,X(u)=x(u),0\le u<s\}=\mathrm{Pr}\{X(t+s)=j\mid X(s)=i\}$$

In other words, the conditional distribution of the future $X(t+s)$ given the present $X(s)$ and the past $X(u)$ depends only on the present and is independent of the past.

If $\mathrm{Pr}\{X(t+s)=j\mid X(s)=i\}$ is independent of $s$, then the CTMC is said to be homogeneous (has stationary transition probabilities).

Interpretation

A CTMC is the continuous-time analog of a discrete-time Markov chain. The key difference is that the process spends a random, exponentially-distributed amount of time in each state before jumping to another state.

Equivalent Definition

A CTMC can be equivalently defined as a stochastic process where each time it enters state $i$:

1. The amount of time spent in state $i$ before making a transition is exponentially distributed with rate ${\nu }_i$.
2. When the process leaves state $i$, it enters state $j$ with probability $P_{ij}$, where $P_{ii}=0$ and ${\sum }_j{P}_{ij}=1$.

The amount of time spent in state $i$ and the next state visited must be independent random variables (otherwise the Markovian property is violated).

Transition Probability Function

The transition probability function is:

$P_{ij}(t)=\mathrm{Pr}\{X(t+s)=j\mid X(s)=i\}$

with $P_{ij}(0)=0$ for $i\ne j$ and $P_{ii}(0)=1$.

Relation to Poisson Process

The Poisson Process is a CTMC where transitions only go from state $n$ to $n+1$ (pure birth process).

Related

• Discrete-time Markov Chain
• Markov Property
• Pure Birth Process
• Birth and Death Process

Exercises

Back to Roadmap 📖 → 🃏 → ✏

Kuis 2 2025 No. 1. Suatu reaksi kimia mengubah molekul A menjadi B secara irreversibel. Awalnya ada $N$ molekul A. Jika pada waktu $t$ terdapat $j$ molekul A, setiap molekul berubah menjadi B dalam $[t,t+h)$ dengan probabilitas $qh+o(h)$, untuk $q>0$. Modelkan banyaknya molekul A sebagai CTMC.

Tentukan state-space (ruang keadaan).

Jawaban: $\{0,1,\dots ,N\}$. Karena awalnya ada $N$ molekul dan reaksi irreversibel (hanya berkurang), jumlah molekul A hanya bisa bernilai $0$ sampai $N$.