Embedded Markov Chain

Definition

Given a CTMC $\{Y(t):t\ge 0\}$ on state space $S$, the Embedded Markov Chain (or jump process) is the discrete-time process $\{X_n=Y(T_n):n\in {\mathbb{Z}}^{+}\}$ where $T_n$ is the sojourn time between the $(n-1)$-th and $n$-th events.

The embedded chain is obtained by sampling the CTMC at the jump times (ignoring the time spent in each state).

Jump Transition Probabilities

For $i,j\in S$:

$P_{ij}=\mathrm{Pr}\{X_{n+1}=j\mid X_n=i\}=\mathrm{Pr}\{Y(T_{n+1})=j\mid Y(T_n)=i\}$

where $P_{ij}$ are exactly the entries of the P-matrix from the Q-matrix specification: $P_{ij}=q_{ij}/{\nu }_i$ for $i\ne j$, and $P_{ii}=0$.

Key Properties

• The embedded chain is a homogeneous discrete-time Markov chain with countable state space
• $P_{ii}=0$ for all $i$ — states do not transition to themselves (a CTMC always changes state at jump times)
• Periodicity is not possible in CTMC

Interpretation

The embedded chain captures the sequence of states visited, ignoring how long the process stays in each state. It provides the discrete-time “skeleton” of the CTMC. The CTMC can be viewed as an embedded DTMC with exponentially distributed holding times.

Example: Poisson Process

For a Poisson process $\{N(t):t\ge 0\}$ with rate $\lambda$, the embedded MC has:

$P_{i,i+1}=1,\forall i$

It only jumps from $i$ to $i+1$ (pure birth process).

Relation to Limiting Probabilities

The limiting probabilities of the CTMC are related to those of the embedded chain:

$P_i=\frac{{\pi }_i/{\nu }_i}{{\sum }_j{\pi }_j/{\nu }_j}$

where $P_i$ are CTMC limiting probabilities, ${\pi }_i$ are embedded DTMC limiting probabilities, and ${\nu }_i$ are transition rates.

Related

• Continuous-Time Markov Chain
• Transition Rate Matrix
• Class Properties of CTMC

Exercises

Back to Roadmap 📖 → 🃏 → ✏

Kuis 2 2025 No. 4. Reaksi kimia: $N$ molekul A → B, laju $qj$ dari state $j$. Tentukan $P_{j,j-1}$ dan $P_{j,j}$ (probabilitas transisi embedded MC).

Jawaban: $P_{j,j-1}=1$ untuk $j>0$, $P_{0,0}=1$ (absorbing). Karena hanya satu transisi yang mungkin ($j\to j-1$), probabilitas embedded MC untuk transisi tersebut adalah 1. $P_{jj}=0$ untuk semua state non-absorbing (embedded MC selalu berubah state).

Dari Q ke P. Diberikan $Q=\left[\begin{array}{ccc}-2& 2& 0\\ 1& -3& 2\\ 0& 2& -2\end{array}\right]$. Tentukan matriks transisi embedded MC $P$.

Jawaban: ${\nu }_0=2$, $P_{01}=1$. ${\nu }_1=3$, $P_{10}=1/3$, $P_{12}=2/3$. ${\nu }_2=2$, $P_{21}=1$. Jadi $P=\left[\begin{array}{ccc}0& 1& 0\\ 1/3& 0& 2/3\\ 0& 1& 0\end{array}\right]$.