Stochastic Model

• italic: Definition / About
• Bold: Theorem / Property
• Standard: Procedure / Example / Cheatsheet

Prerequisites

• Probability and Random Variables Review
• Conditional Probability and Expectation Review

Stochastic Process Fundamentals

• Stochastic Process
• State (Stochastic)
  • State vs Sample
• Markov Property
• Markov Process

Markov Chain Fundamentals

• Discrete-time Markov Chain
• Transition Probability
• Transition Probability Matrix
• n-step Transition Matrix
• Chapman-Kolmogorov Equation
  • n-step Transition Matrix Identity

State Properties

• Accessible State

• Communication

• Communicating Classes

• Irreducible

• Recurrent vs Transient

• Period (Stochastic)

  • Aperiodic vs Periodic State

• Absorbing State

• State Classification Cheatsheet

Limiting Behavior

• Limiting Probability
• Regular Transition Probability Matrix
  • Regular Chain is Irreducible and Aperiodic
• Stationary Distribution
• Ergodic State
  • Positive Recurrent State
• Limiting Distribution Theorem

Transient Analysis

• Mean Time Spent in Transient States
• Hitting Probability (Transient)
• First Step Analysis
  • Limiting vs Hitting Probability

Special Topics

• Random Walk Markov Chain
• Branching Process
• Extinction Probability
• Time Reversible Markov Chain
  • Kolmogorov’s Criterion
• Markov Chain Monte Carlo

Counting and Poisson Processes

• Counting Process
  • Independent and Stationary Increments
• Poisson Process
  • Sum of Poisson Random Variables
  • Poisson-Binomial Distribution
  • Compound Poisson Process
  • Nonhomogeneous Poisson Process
  • Mixed Poisson Process
• Inter-arrival Times
• Waiting Times
• Conditional Distribution of Arrival Times
• Thinning
  • Independence of Thinned Processes

Related Distributions

• Poisson Distribution
• Exponential Distribution
  • Memoryless Property
  • Failure Rate Function
• Hyperexponential Distribution
• Hypoexponential Distribution
• Coxian Distribution

Cheatsheets

• Poisson Process Cheatsheet
• Stochastic State Relationships Cheatsheet

Examples

• Urn Ball Replacement n-step Transition
• Exponential Random Variables and Expected Discounted Returns
• Time Spent in a Bank
• Automobile Accident Damage
• Commodity Ordering
• Analyzing Greedy Algorithms for the Assignment Problem
• Cells in the Body
• Customers in Line
• Function o(h)
• Immigrants into a Territory
• Nonnegative Offers
• System with Classified Individuals
• The Coupon Collecting Problem
• An Infinite Server Queue
• Minimizing the Number of Encounters
• Tracking the Number of HIV Infections
• Insurance Claims
• An Optimization Example
• Nonhomogeneous Poisson Process Demand of a Facility
• Bugs in Operating Time
• Siegbert Hot Dog Stand
• The Output Process of an Infinite Server Poisson Queue
• Families Migrate to an Area
• Busy Periods in Single Server Poisson Arrival Queues
• Gamma Density
• Insurance Company Policyholders
• Transition Matrix Example
• n-step Transition Matrix Example
• Gambling Model
• First Step Analysis - Gambler’s Ruin
• Social Class Problem Example
• Random Walk Model Example