5.1 Convergence in Probability

<< 4.4 Order Statistics | 5.2 Convergence in Distribution.md >>

To emphasize the fact that we are working with sequences of random variables, we may place a subscript $n$ on the appropriate random variable, e.g., write sequence of $\bar{X}$ as ${\bar{X}}_n$

Definition 5.1.1: Convergence in probability

Definition

Let

• $\{X_n\}$ : Sequence of Random variables
• $X$ : Random variable

If $ \lim_{ n \to \infty } P[|X_{n}-X|\geq \epsilon] = 0, \quad \forall \epsilon>0$$

• Or equivalently $ \lim_{ n \to \infty } P[|X_{n}-X| < \epsilon] = 1, \quad \forall \epsilon>0 $$

Then

• We say $\{X_n\}$ converges in probability to $X$
• We write $ X_{n} \xrightarrow P X $$

Theorem 5.1.1: Weak law of large numbers

Theorem

Let

• $\{X_n\}$ : Sequence of random samples, with
  • Common mean $\mu$
  • Common variance ${\sigma }^2<\infty$
• $\bar{X}{n}=\frac{1}{n}\sum{i=1}^nX_{i}$$

Then

$${\bar{X}}_n\stackrel{P}{\to }\mu$$

In the following sections (Theorem 5.1.2 to Theorem 5.1.5) we describe some theorems related to convergence of sequence of random variables. For brevity, we implicitly let:

• $X,Y$ : Random variables
• $X_n,Y_n$: Sequences of $X,Y$ respectively
• $a\in \mathbb{R}$ : Some constant

Theorem 5.1.2

Suppose

• $X_n\stackrel{P}{\to }X$
• $Y_n\stackrel{P}{\to }Y$

Then

$$X_n+Y_n\stackrel{P}{\to }X+Y$$

Theorem 5.1.3

$$X_n\stackrel{P}{\to }X⟹a{X}_n\stackrel{P}{\to }aX$$

Theorem 5.1.4

Let

• $g:\mathbb{R}\to \mathbb{R}$

Suppose

• $X_n\stackrel{P}{\to }a$
• $g$ is continuous at point $a$

Then

$$g(X_n)\stackrel{P}{\to }g(a)$$

Theorem 5.1.5

Suppose

• $X_n\stackrel{P}{\to }X$
• $Y_n\stackrel{P}{\to }Y$

Then

$$X_n{Y}_n\stackrel{P}{\to }XY$$

Definition 5.1.2: Consistent estimator

Definition

Let

• $X$ : Random variable, with
  • $\Omega$ : Parameter space
  • cdf $F(x;\theta \in \Omega )$
• $X_1,\dots ,X_n$ : Random sample of $X$
• $T_n$ : Statistic

If

$$T_n\stackrel{P}{\to }\theta$$

Then we say $T_n$ is a consistent estimator of $\theta$

Theorem: Law of large numbers for sample variance

Let

• $X_i$ : Sequence of random samples, with
  • Common mean $\mu$
  • Common variance ${\sigma }^2<\infty$
• $S_n^2=\frac{1}{n-1}{\sum }_{i=1}^n(X_i-{\bar{X}}_n{)}^2$ : Sample variance

Then $S_n^2\stackrel{P}{\to }{\sigma }^2$

Note

This theorem states that the sample variance $S_n^2$ is a consistent estimator of the population variance ${\sigma }^2$.

Before stating the strong law of large numbers, we need to introduce the concept of almost sure convergence, which is a stronger form of convergence than convergence in probability.

Definition: Almost sure convergence

Definition

Let

• $X_n$ : Sequence of Random variables
• $X$ : Random variable

If $ P\left[\lim_{n \to \infty} X_{n} = X\right] = 1 $$

Then

• We say $X_n$ converges almost surely to $X$
• We write $ X_{n} \xrightarrow{a.s.} X $$

Theorem: Strong law of large numbers

Let

• $X_n$ : Sequence of random samples, with
  • Common mean $\mu$
  • Common variance ${\sigma }^2<\infty$
• ${\bar{X}}_n=n^{-1}{\sum }_{i=1}^n{X}_i$

Then ${\bar{X}}_n\stackrel{a.s.}{\to }\mu$

The strong law of large numbers provides a stronger guarantee than the weak law of large numbers.

While the weak law states that ${\bar{X}}_n$ converges to $\mu$ in probability, the strong law states that ==${\bar{X}}_n$ converges to $\mu$ almost surely==, meaning that with probability 1, the sample mean will eventually stabilize around the true mean $\mu$ as $n\to \infty$.

The key difference between strong and weak law of large numbers are:

• Weak: For any $ϵ>0$, the probability that ${\bar{X}}_n$ is “far” from $\mu$ becomes small as $n$ gets large
• Strong: With probability $1$, the sequence ${\bar{X}}_n$ will converge to $\mu$