6.2 Rao-Cramér Lower Bound and Efficiency

<< 6.1 Maximum Likelihood Estimation.md | 6.3 Maximum Likelihood Tests >>

Assumptions 6.2.1: Additional regularity conditions 1

• $(R3)$ : The pdf $f(x;\theta )$ is twice differentiable as a function of $\theta$.
• $(R4)$ : The integral $\int f(x;\theta )$ can be differentiated twice under the integral sign as a function of $\theta$.

Definition: Score function

Definition

Let

• $X$ : Random variable, with
  • pdf $f(x;\theta )$, for $\theta \in \Omega$

Then the score function is defined as $S(x;\theta) = \frac{\partial}{\partial \theta} \ln f(x;\theta)$$

Definition: Fisher information

Definition

Let

• $X$ : Random variable, with
  • pdf $f(x;\theta )$, for $\theta \in \Omega$
• $S(X;\theta )$ : Score Function

Then the Fisher information is defined as:

$$\begin{array}{rl}I(\theta )& =E\left[S(X;\theta {)}^2\right]\\ & =E\left[{\left(\frac{\partial }{\partial \theta }\ln f(X;\theta )\right)}^2\right]\\ & =-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\ln f(X;\theta )\right]\end{array}$$

Theorem 6.2.1: Rao-Cramér lower bound

Definition

Let

• $X_1,\dots ,X_n$: Random sample, with pdf $f(x;\theta ),\theta \in \Omega$
• $Y=u(X_1,X_2,\dots ,X_n)$ : Statistic
• $k(\theta )=E(Y)$[^1]
• $I(\theta )$ : Fisher information

Assume regularity conditions and additional regularity conditions 1 hold.

Then $ \operatorname{Var}(Y)\geq \boxed{\frac{[k’(\theta)]^2}{nI(\theta)}} $$

• We say $[k^{\prime }(\theta ){]}^2/nI(\theta )$ is the Rao-Cramer lower bound of $Y$

Corollary 6.2.1: Rao-Cramér bound for unbiased estimators

Under the same conditions as Theorem 6.2.1 Rao-Cramér lower bound

If $Y$ is an unbiased estimator of $\theta$ (so that $k(\theta )=\theta$ and $k^{\prime }(\theta )=1$)

Then $\mathrm{Var}(Y)\ge \frac{1}{nI(\theta )}$

Important

We call the ratio of Rao-Cramer Lower Bound divided by $\mathrm{Var}(Y)$ the efficiency of $Y$: $\frac{\frac{[k^{\prime }(\theta ){]}^2}{nI(\theta )}}{\mathrm{Var}(Y)}$ This means

1. If efficiency = 1, then $Y$ is efficient
2. If efficiency approaches 1, then $Y$ is asymptotically efficient

Thus $Y$ is efficient if and only if $\mathrm{Var}(Y)=\frac{1}{nI(\theta )}⟺nI(\theta )\cdot \mathrm{Var}Y=1$ Assumptions 6.2.1 Additional regularity conditions 1 holds

Definition 6.2.1: Efficient estimator

Definition

Let $Y$ : Unbiased estimator of parameter $\theta$

Then $Y$ is an efficient estimator $⟺$ $Y$ attains the Rao-Cramér lower bound

Assumptions 6.2.2: Additional regularity conditions 2

• $(R5)$ : The pdf $f(x;\theta )$ is three times differentiable as a function of $\theta$. Further, for all $\theta \in \Omega$, there exist a constant $c$ and a function $M(x)$ such that $\left|\frac{{\partial }^3}{\partial {\theta }^3}\log f(x;\theta )\right|\le M(x)$ with $E_{{\theta }_0}[M(X)]<\infty$, for all ${\theta }_0-c<\theta <{\theta }_0+c$ and all $x$ in the support of $X$.

Theorem 6.2.2

Assume

• $X_1,\dots ,X_n$ : Random samples, with
  • pdf $f(x;{\theta }_0)$, for ${\theta }_0\in \Omega$
  • All regularity conditions $(R0)-(R5)$ are satisfied.

If fisher information satisfies $0<I({\theta }_0)<\infty$

Then any consistent sequence of solutions of the mle equations satisfies

$$\sqrt{n}(\hat{\theta }-{\theta }_0)\stackrel{D}{\to }N\left(0,\frac{1}{I({\theta }_0)}\right)$$