Stochastic State Relationships

Here is a summary of the relationships between the core concepts in stochastic models and Markov chains, particularly focusing on state classification and long-term behavior.

Core Property Implications

The following visual map shows how different properties of Markov chains and their states logically imply one another.

graph TD
    Regular["Regular TPM (P^k > 0)"] -->|Implies| Irreducible["Irreducible (All states communicate)"]
    Regular -->|Implies| Aperiodic["Aperiodic (Period d = 1)"]
    
    Irreducible --> Finite{"Finite State Space?"}
    Finite -->|Yes| PosRecurrent["Positive Recurrent (Finite expected return time)"]
    
    PosRecurrent --> Ergodic["Ergodic State"]
    Aperiodic --> Ergodic
    
    Ergodic -->|Requires Irreducible chain| LimDist["Limiting Distribution Exists (π_j = lim P^n_ij)"]
    Irreducible -->|Requires Ergodic chain| StatDist["Unique Stationary Distribution Exists (π = πP)"]
    
    LimDist -->|Always equals| StatDist
    StatDist -.->|May exist without| LimDist
    
    Periodic["Periodic Chain"] -.->|Can have| StatDist
    Periodic -.->|Cannot have| LimDist

Concept Summaries

Regular vs Irreducible vs Aperiodic

• Regular TPM: A transition matrix where some power ${\mathbf{P}}^k$ has all strictly positive entries.
  • Implies: The chain is both irreducible and aperiodic (Theorem).
• Irreducible: All states communicate with each other (one single communicating class).
• Aperiodic: The period of a state is $d=1$, meaning you can return to it at irregular times (no fixed cycle).

Ergodic States

An Ergodic State is a state that is both:

1. Positive Recurrent: The expected time to return to the state is finite.
2. Aperiodic.

Note: In a finite irreducible Markov chain, all states are positive recurrent. If the chain is also aperiodic, all states are ergodic.

Limiting vs Stationary Distributions

• Limiting Distribution: The long-run probability of being in a state, independent of the starting state (${\pi }_j={\lim }_{n\to \infty }{P}_{ij}^n$).
  • Requires: An irreducible, ergodic (which implies aperiodic) Markov chain (Theorem).
  • Always: If it exists, it is always a stationary distribution.
• Stationary Distribution: A probability vector $\pi$ such that $\pi =\pi \mathbf{P}$. If you start with this distribution, you stay in it.
  • May Exist Without: A stationary distribution can exist even if a limiting distribution does not (e.g., in periodic chains where probabilities cycle and do not converge to a single limit).
  • Uniqueness: For an irreducible ergodic chain, there is a unique stationary distribution, which is equal to the limiting distribution.