4.2 Confidence Intervals

<< 4.1 Sampling and Statistics.md | 4.2.1 Confidence Intervals for Difference in Means >>

Definition 4.2.1: Confidence interval

Let

• $X$ : Random variable
• $X_1,\dots ,X_n$ : Random sample of $X$, with
  • pdf $f(x;\theta ),\theta \in \Omega$
• $0<\alpha <1$
• $L=L(X_1,\dots ,X_n)$ : Statistic
• $U=U(X_1,\dots ,X_n)$ : Statistic

If $1-\alpha =P_{\theta }[\theta \in (L,U)]$

Then

• We say interval $(L,U)$ is a $(1-\alpha )100\%$ confidence interval for $\theta$
• The probability that the interval $(L,U)$ includes $\theta$ is $1-\alpha$, which is called the confidence coefficient/level of the interval.
• If $E_{\theta }(U_1-L_1)\le E_{\theta }(U_2-L_2),\forall \theta \in \Omega$, then we ay $(L_1,U_1)$ is more efficient than $(L_2,U_2)$

Link to original

Theorem 4.2.1: Central limit theorem

Let

• $X_1,\dots ,X_n$ : Observations of Random sample, with
  • Mean $\mu$
  • Finite variance ${\sigma }^2$
• $\Phi$ : Distribution function of $N(0,1)$
• $W_n=\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}$

Then

• We say the distribution function of $W_n$ converges to $\Phi$ as $n\to \infty$
• We write $W_n\stackrel{D}{\to }N(0,1)$

---

There are several procedures for obtaining confidence intervals. We will discuss one based on a pivot random variable in this article.

The pivot variable is usually a function of an estimator of $\theta$. The parameter and the distribution of the pivot is known.

In general, the examples we will show below for finding the confidence interval for a parameter $\theta$ is summarized in these steps:

1. Find estimator for $\theta$, e.g., $\hat{\theta }=\bar{X}$ or $\hat{\theta }=S^2$
2. Find pivot variable $Z$ which is a function of $\hat{\theta }$, with distribution and parameter(s) of the distribution known
3. Compute the confidence interval $1-\alpha =P_{\mu }[\Phi <Z(\hat{\theta })<-\Phi ]$
4. Find the realization of the confidence interval

Example 4.2.1: Confidence interval for $\mu$ under normality

Let

• $X_1,\dots ,X_n$ : Random sample, with
  • distribution $N(\mu ,{\sigma }^2)$
  • $\bar{X}$ : Sample mean
  • $S^2$ : Sample variance

Recall from 4.1 Sampling and Statistics.md:

• $\bar{X}$ : MLE of $\mu$
• $\frac{n-1}{n}{S}^2$ : MLE of ${\sigma }^2$

By Normal Distribution Relationships, $T=\frac{\bar{X}-\mu }{S/\sqrt{n}}\sim t(n-1)$ This random variable $T$ is our pivot variable.

Let $0<\alpha <1$. We define $t_{\alpha /2,n-1}$ to be the upper $\alpha /2$ critical point of a $t$- distribution with $n-1$ degrees of freedom; i.e., $\alpha /2=P(T>t_{\alpha /2,n-1})$. We can obtain

1-\alpha & = P(-t_{\alpha/2,n-1}<T<t_{\alpha/2,n-1}) \\ & = P_{\mu}\left( -t_{\alpha/2,n-1}< \textcolor{yellow}{\frac{\bar{X}-\mu}{S^2/\sqrt{ n }}} <t_{\alpha/2,n-1} \right) \\ & = P_{\mu}\left( -t_{\alpha/2,n-1} \textcolor{yellow}{\frac{S^2}{\sqrt{ n }}} < \bar{X}-\mu <t_{\alpha/2,n-1} \textcolor{yellow}{\frac{S^2}{\sqrt{ n }}} \right) \\ & = P_{\mu}\left( t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} \textcolor{yellow}{> \mu-\bar{X}>-}t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} \right) \\ & = P_{\mu}\left( \textcolor{yellow}{-}t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} < \textcolor{yellow}{\mu-\bar{X}} <t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} \right) \\ & = P_{\mu}\left( \textcolor{yellow}{\bar{X}} -t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} < \mu < \textcolor{yellow}{\bar{X}} + t_{\alpha/2,n-1} \frac{S^2}{\sqrt{ n }} \right) \\ \end{align} $$ Once the sample is drawn, let $\bar{x},s$ denote realized values of $\bar{X}$ and $S$, respectively. Then $(1-\alpha)100\%$ [[3 Reference/def-confidence-interval_202507220823\|confidence interval]] for $\mu$ is

\left( \bar{x} -t_{\alpha/2,n-1} \frac{s}{\sqrt{ n }} < \mu < \bar{x} +t_{\alpha/2,n-1} \frac{s}{\sqrt{ n }} \right)

Also, we often say - This interval is the **$(1-\alpha)100\%$ $t$-interval for $\mu$**. - The estimate of $s/\sqrt{ n }$ (standard deviation of $\bar{X}$) is the **standard error** of $\bar{X}$ ## Example 4.2.2: Large sample confidence interval for $\mu$ Let - $X$ : [[3 Reference/Def-random-variable\|Random variable]], with - [[3 Reference/Def-mean\|Mean]] $\mu$ - [[3 Reference/Def-variance\|Variance]] $\sigma^2$ - $X_{1},\dots,X_{n}$ : [[3 Reference/Def-random-sample\|Random sample]], with - $\bar{X}$ : [[3 Reference/common-distribution-equations_202507221712#sample-mean-and-variance-distributions\|Sample Mean]] - $S^2$ : [[3 Reference/common-distribution-equations_202507221712#sample-mean-and-variance-distributions\|Sample Variance]] From [[3 Reference/mathstat5.3#theorem-531-central-limit-theorem\|Theorem 5.3.1 Central limit theorem]], we know that $$ Z_{n} = \frac{\bar{X}-\mu}{S/\sqrt{ n }} \xrightarrow{D} N(0,1).$$ Let $z_{\alpha/2}$ be the upper $\alpha/2$ critical point of [[3 Reference/Continuous Distributions#z-distribution\|Z-distribution]]. Similar to [[3 Reference/4.2-confidence-intervals_202507220822#example-421-confidence-interval-for--mu-under-normality\|Example 4.2.1]], we can obtain

\begin{align} 1-\alpha & \approx P_{\mu}\left( -z_{\alpha/2}< \frac{\bar{X}-\mu}{S/\sqrt{ n }} < z_{\alpha/2}\right) \ & \approx P_{\mu}\left( \bar{X} - z_{\alpha/2} \frac{S}{\sqrt{ n }} < \mu < \bar{X} + z_{\alpha/2} \frac{S}{\sqrt{ n }} \right) \end{align}

And the **approximate** $(1-\alpha)100\%$ [[3 Reference/def-confidence-interval_202507220823\|confidence interval]] for $\mu$ is given by $$\left( \bar{x} -z_{\alpha/2} \frac{s}{\sqrt{ n }} < \mu < \bar{x} +z_{\alpha/2} \frac{s}{\sqrt{ n }} \right)$$ This interval is called the **large sample confidence interval** for $\mu$. ## Example 4.2.3: Confidence interval for proportion Let - $X$ : [[3 Reference/Def-random-variable\|Random variable]], with - $X\sim \text{Bernoulli}(p)$, from [[3 Reference/Discrete Distributions#bernoulli-distribution\|Bernoulli distribution]] - $X_{1},\dots,X_{n}$ : [[3 Reference/Def-random-sample\|Random sample]] of $X$ - $\hat{p}=\bar{X}$ : Sample proportion of success Then - $\operatorname{Var}(X_{i})=p(1-p)$ Let $\hat{p}=\frac{1}{n}\sum_{i=1}^nX_{i}$ : Sample average. We can then get

\begin{align} \operatorname{Var}(\hat{p}) & = \operatorname{Var}\left( \frac{1}{n}\sum X_{i} \right) \ & = \frac{1}{n} \operatorname{Var}\left( \sum X_{i} \right) \ & = \frac{1}{n}\sum \operatorname{Var}(X_{i})\quad X_{i}\text{ random samples} \ & = \frac{1}{n} p(1-p) \end{align}

From [[3 Reference/4.2-confidence-intervals_202507220822#theorem-421-central-limit-theorem\|CLT]], we have

\begin{align} Z & = \frac{\bar{X}-\mu}{\sigma^2/\sqrt{ n }} \ & = \frac{\hat{p}-p}{\sqrt{ p(1-p)/n }}, \quad \left(\begin{array}{m} \bar{X}=\hat{p} \ \mu = p \ \sigma^2= p(1-p) \end{array} \right) \end{align}

and that $Z \xrightarrow{D}N(0,1)$. By [[3 Reference/mathstat5.1#theorem-law-of-large-numbers-for-sample-variance\|Theorem Law of large numbers for sample variance]], we replace $p(1-p)$ with its estimate $\hat{p}(1-\hat{p})$. Then, similar to [[3 Reference/4.2-confidence-intervals_202507220822#example-422-large-sample-confidence-interval-for--mu\|Example 4.2.2,]] an **approximate** $(1-\alpha)100\%$ [[3 Reference/def-confidence-interval_202507220823\|confidence interval]] for $p$ is $$ \left( \hat{p}-z_{\alpha/2} \sqrt{ \hat{p}(1-\hat{p})/n } ,\;\; \hat{p}+z_{\alpha/2} \sqrt{ \hat{p}(1-\hat{p})/n } , \right) $$ where $\sqrt{ \hat{p}(1-\hat{p})/n }$ is the **standard error** of $\hat{p}$