Tugas 1

Question 1

1.a.

A Markov chain is a stochastic process where the future state depends only on the current state, not on the past states.¹

Let

• $X_n$ : discount level of policyholder after $n$ years
• $S=\{0,1,2,3\}$ where: - State 0: no discount (0%) - State 1: 15% discount - State 2: 25% discount - State 3: 40% discount
• $T=\{0,1,2,\dots \}$ representing years

$X_n$ represents the premium discount status at the end of year $n$

$\therefore$ The Markov property holds because next year’s discount depends only on the current year’s discount and whether a claim was made.

1.b.

From Rangkuman:

• Transition probability $P_{ij}$ is the probability of transitioning from state $i$ to state $j$ in one time step.
• Transition matrix $\mathbf{P}$ contains all $P_{ij}$ in matrix form.

Given:

• $P(\text{no claim})=\frac{3}{4}$
• $P(\text{at least one claim})=\frac{1}{4}$

Then:

• From state 0 (no discount): with no claim → state 1; with claim → state 0
• From state 1 (15%): with no claim → state 2; with claim → state 0
• From state 2 (25%): with no claim → state 3; with claim → state 1
• From state 3 (40%): with no claim → state 3; with claim → state 2

$\therefore$ Transition Matrix:

$$\mathbf{P}=\left[\begin{array}{cccc}\frac{1}{4}& \frac{3}{4}& 0& 0\\ \frac{1}{4}& 0& \frac{3}{4}& 0\\ 0& \frac{1}{4}& 0& \frac{3}{4}\\ 0& 0& \frac{1}{4}& \frac{3}{4}\end{array}\right]$$

1.c.

We need: $P\{X_3=3\mid X_0=0\}=P_{03}^{(3)}$

From Chapman-Kolmogorov:

• ${\mathbf{P}}^{(n)}={\mathbf{P}}^n$
• $P_{ij}^{(n+m)}={\sum }_k{P}_{ik}^{(n)}{P}_{kj}^{(m)}$

Using Python to compute $P^{(3)}$:

$\therefore$ $P_{03}^{(3)}=0.421875$

1.d.

We need: $P\{X_5=2\mid X_4=3\}$

This is a one-step transition from state 3 to state 2.

$P_{ij}$ represents the probability of transitioning from state $i$ to state $j$ in one time step.¹

From the transition matrix in 1.b.:

$$P_{32}=\frac{1}{4}=0.25$$

$\therefore$ $P\{X_5=2\mid X_4=3\}=0.25$

Question 2

Given:

• State space: $S=\{0,1,2,3\}$
• Transition matrix:

$$\mathbf{P}=\left[\begin{array}{cccc}0.4& 0.3& 0.2& 0.1\\ 0.1& 0.4& 0.3& 0.2\\ 0.3& 0.2& 0.1& 0.4\\ 0.2& 0.1& 0.4& 0.3\end{array}\right]$$

• Initial distribution: ${\pi }^{(0)}=(0.25,0.25,0.25,0.25)$

2.a.

From Chapman-Kolmogorov:

• The distribution at time $n$ is: ${\pi }^{(n)}={\pi }^{(0)}{\mathbf{P}}^n$

We need the probability of being in state 0 at time 2 (${\pi }^{(2)}={\pi }^{(0)}{\mathbf{P}}^2$)

$\therefore$ $\mathrm{Pr}(X_2=0)=0.25$

2.b.

We need the probability of being in state 2 at time 3 (${\pi }^{(3)}={\pi }^{(0)}{\mathbf{P}}^3$)

$\therefore$ $\mathrm{Pr}(X_3=2)=0.25$

Question 3

Given transition matrix:

$$\mathbf{P}=\left[\begin{array}{cccccc}\frac{1}{3}& \frac{1}{3}& \frac{1}{3}& 0& 0& 0\\ \frac{1}{2}& \frac{1}{4}& \frac{1}{4}& 0& 0& 0\\ 0& 0& 0& \frac{1}{4}& \frac{1}{4}& \frac{1}{2}\\ \frac{1}{4}& \frac{1}{4}& \frac{1}{4}& 0& 0& \frac{1}{4}\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0& 1\end{array}\right]$$

We need to classify which states are transient and which are recurrent. (See Recurrent vs Transient)

By definition:

Accessible State:

State $j$ is accessible from state $i$ if $P_{ij}^{(n)}>0$ for some $n\ge 0$.

Communication:

• Two states communicate if they are mutually accessible.
• Communication satisfies: reflexive, symmetric, transitive (equivalence relation).

Communicating Classes:

• States that communicate form a class.
• Classes are either identical or disjoint.
• A chain is irreducible if there is only one class.

First Step Analysis:

• From state 0: Can go to states 0, 1, 2
• From state 1: Can go to states 0, 1, 2
• From state 2: Can go to states 3, 4, 5
• From state 3: Can go to states 0, 1, 2, 5
• From state 4: Can go to state 2
• From state 5: Can only go to state 5 (absorbing)

So:

1. {0, 1}:
   • 0 → 1 (via $P_{01}=\frac{1}{3}>0$)
   • 1 → 0 (via $P_{10}=\frac{1}{2}>0$)
2. {2, 3, 4}:
   • 2 → 3 → 4 → 2 forms a cycle
3. {5}:
   • State 5 is absorbing ($P_{55}=1$)
   • It’s a singleton class.

$\therefore$ Therefore:

• Transient states: 0, 1, 2, 3, 4
• Recurrent states: 5

Question 4

Given:

• Demand distribution: $\mathrm{Pr}({\xi }_n=0)=0.1$, $\mathrm{Pr}({\xi }_n=1)=0.3$, $\mathrm{Pr}({\xi }_n=2)=0.3$, $\mathrm{Pr}({\xi }_n=3)=0.2$, $\mathrm{Pr}({\xi }_n=4)=0.1$
• Policy: $(s,S)$ with $s=1$ and $S=4$

So:

• If inventory after demand ≤ s (= 1), order up to S (= 4)
• If inventory after demand > s (= 1), no order

4.a.

$P_{41}$ = Probability of transitioning from state 4 to state 1.

Analysis

• Demand = 0 → Inventory = 4 → Since 4 > s = 1, no order → Next state = 4
• Demand = 1 → Inventory = 3 → Since 3 > 1, no order → Next state = 3
• Demand = 2 → Inventory = 2 → Since 2 > 1, no order → Next state = 2
• Demand = 3 → Inventory = 1 → Since 1 ≤ s, order to 4 → Order 3 units → Next state = 4
• Demand = 4 → Inventory = 0 → Since 0 ≤ s, order to 4 → Order 4 units → Next state = 4

So:

• From state 4, the next states possible are: 4, 3, 2 (all > 1)
• There is no scenario where the next state = 1

$\therefore$ $P_{41}=0$

4.b.

$P_{04}$ = Probability of transitioning from state 0 to state 4.

Analysis

• Demand = 0 → Inventory = 0 → Since 0 ≤ s = 1, order to S = 4 → Order 4 units → Next state = 4
• Demand = 1 → Inventory = 0 → Since 0 ≤ s, order to 4 → Next state = 4
• Demand = 2 → Inventory = 0 → Since 0 ≤ s, order to 4 → Next state = 4
• Demand = 3 → Inventory = 0 → Since 0 ≤ s, order to 4 → Next state = 4
• Demand = 4 → Inventory = 0 → Since 0 ≤ s, order to 4 → Next state = 4

$\therefore$ All scenarios lead to state 4. So, $P_{04}=1.0$

Footnotes

1. Rangkuman ↩ ↩²