Maximum Likelihood Method

About

Maximum Likelihood Method estimates parameters by maximizing the probability of observing the given data, assuming a specific probability distribution for the process.

Core Concept

The likelihood function $L(\theta |Y_1,\dots ,Y_n)$ is the joint probability density viewed as a function of parameters $\theta$, with data fixed.

$L(\theta )=f(Y_1,Y_2,\dots ,Y_n|\theta )$

The maximum likelihood estimator (MLE) is:

$\hat{\theta }=\arg {\max }_{\theta }L(\theta )$

Usually, maximize $\log L(\theta )$ for computational convenience.

Advantages

• Uses all information in the data (not just moments)
• Asymptotically efficient (lowest variance)
• Consistent and asymptotically normal
• Works well for large samples

Limitations

• Requires specification of full distributional form
• Computationally intensive for complex models

Example: AR(1) Model

Assume white noise $e_t\sim \text{NIID}(0,{\sigma }_e^2)$.

Recall for $X\sim N(0,{\sigma }_e^2)$, the pdf is:

$$f(X)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{x^2}{2{\sigma }^2}\right)$$

Step 1: Joint Density of Errors

The AR(1) errors are white noise: $e_t\sim \text{NIID}(0,{\sigma }_e^2)$ for $t=2,3,\dots ,n$. Since they are independent and identically distributed, the joint density equals the product of individual densities.

Derivation:

Recall the pdf of a normal random variable $X\sim N(\mu ,{\sigma }^2)$ is: $f(X)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(X-\mu {)}^2}{2{\sigma }^2}\right)$

Since $e_t$ has mean zero and variance ${\sigma }_e^2$: $f(e_t)=\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\exp \left(-\frac{e_t^2}{2{\sigma }_e^2}\right)$

The joint density is the product of $(n-1)$ independent densities:

$$\begin{array}{rl}f(e_2,e_3,\dots ,e_n)& =\prod _{t=2}^{n}f(e_t)\\ & =\prod _{t=2}^n\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\exp \left(-\frac{e_t^2}{2{\sigma }_e^2}\right)\\ & ={\left(\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\right)}^{n-1}\prod _{t=2}^n\exp \left(-\frac{e_t^2}{2{\sigma }_e^2}\right)\\ & ={\left(\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\right)}^{n-1}\exp \left(-\frac{1}{2{\sigma }_e^2}\sum _{t=2}^n{e}_t^2\right)\end{array}$$

Step 2: Marginal Density of $Y_1$

For a stationary AR(1) process, the first observation $Y_1$ follows the stationary (marginal) distribution. The AR(1) model is: $Y_t=\mu +\varphi Y_{t-1}+e_t$

The stationary distribution is derived by noting that in the long run: $\text{Var}(Y_t)=\text{Var}(Y_{t-1})={\sigma }_Y^2$

From the model equation:

$$\begin{array}{rl}\text{Var}(Y_t)& ={\varphi }^2\text{Var}(Y_{t-1})+\text{Var}(e_t)\\ {\sigma }_Y^2& ={\varphi }^2{\sigma }_Y^2+{\sigma }_e^2\\ {\sigma }_Y^2(1-{\varphi }^2)& ={\sigma }_e^2\\ {\sigma }_Y^2& =\frac{{\sigma }_e^2}{1-{\varphi }^2}\end{array}$$

The stationary mean is ${\mu }_Y=\frac{\mu }{1-\varphi }$. Thus: $Y_1\sim N\left(\frac{\mu }{1-\varphi },\frac{{\sigma }_e^2}{1-{\varphi }^2}\right)$

Derivation:

Substituting into the normal pdf with mean ${\mu }_Y=\frac{\mu }{1-\varphi }$ and variance ${\sigma }_Y^2=\frac{{\sigma }_e^2}{1-{\varphi }^2}$:

$$\begin{array}{rl}f(Y_1)& =\frac{1}{\sqrt{2\pi \cdot \frac{{\sigma }_e^2}{1-{\varphi }^2}}}\exp \left(-\frac{{\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2}{2\cdot \frac{{\sigma }_e^2}{1-{\varphi }^2}}\right)\\ & =\frac{\sqrt{1-{\varphi }^2}}{\sqrt{2\pi {\sigma }_e^2}}\exp \left(-\frac{(1-{\varphi }^2){\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2}{2{\sigma }_e^2}\right)\end{array}$$

Step 3: Complete Likelihood

The complete likelihood is derived using the chain rule of probability (multiplication rule). This factorizes the joint density of all observations.

Derivation:

By the definition of conditional probability: $P(A,B)=P(A)\cdot P(B|A)$

Applying this to the joint density of observations:

$$\begin{array}{rl}f(Y_1,Y_2,\dots ,Y_n|\varphi ,\mu ,{\sigma }_e^2)& =f(Y_1|\varphi ,\mu ,{\sigma }_e^2)\cdot f(Y_2,\dots ,Y_n|Y_1,\varphi ,\mu ,{\sigma }_e^2)\end{array}$$

Since $e_t=Y_t-\mu -\varphi Y_{t-1}$ for $t\ge 2$, the distribution of future errors depends only on $Y_1$ through the deterministic relationship, not through stochastic dependence. Therefore: $f(e_2,\dots ,e_n|Y_1,\varphi ,\mu ,{\sigma }_e^2)=f(e_2,\dots ,e_n|\varphi ,\mu ,{\sigma }_e^2)$

The complete likelihood is: $L(\varphi ,\mu ,{\sigma }_e^2)=f(Y_1)\cdot f(e_2,\dots ,e_n|Y_1)$

Step 4: Log-Likelihood

The log-likelihood is obtained by combining Steps 1, 2, and 3, and taking the natural logarithm.

Derivation:

First, multiply the marginal density $f(Y_1)$ from Step 2 with the joint error density from Step 1:

$$\begin{array}{rl}L(\varphi ,\mu ,{\sigma }_e^2)& =f(Y_1)\cdot f(e_2,\dots ,e_n)\\ & =\frac{\sqrt{1-{\varphi }^2}}{\sqrt{2\pi {\sigma }_e^2}}\exp \left(-\frac{(1-{\varphi }^2){\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2}{2{\sigma }_e^2}\right)\\ & \cdot {\left(\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\right)}^{n-1}\exp \left(-\frac{1}{2{\sigma }_e^2}\sum _{t=2}^n{e}_t^2\right)\end{array}$$

Taking the natural logarithm:

$$\begin{array}{rl}\log L& =\log \left(\frac{\sqrt{1-{\varphi }^2}}{\sqrt{2\pi {\sigma }_e^2}}\right)-\frac{(1-{\varphi }^2){\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2}{2{\sigma }_e^2}\\ & +(n-1)\log \left(\frac{1}{\sqrt{2\pi {\sigma }_e^2}}\right)-\frac{1}{2{\sigma }_e^2}\sum _{t=2}^n{e}_t^2\\ & =\frac{1}{2}\log (1-{\varphi }^2)-\frac{1}{2}\log (2\pi {\sigma }_e^2)-\frac{(1-{\varphi }^2){\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2}{2{\sigma }_e^2}\\ & -\frac{n-1}{2}\log (2\pi {\sigma }_e^2)-\frac{1}{2{\sigma }_e^2}\sum _{t=2}^n{e}_t^2\\ & =-\frac{n}{2}\log (2\pi {\sigma }_e^2)+\frac{1}{2}\log (1-{\varphi }^2)-\frac{S(\varphi ,\mu )}{2{\sigma }_e^2}\end{array}$$

Where the unconditional sum of squares is: $S(\varphi ,\mu )=(1-{\varphi }^2){\left(Y_1-\frac{\mu }{1-\varphi }\right)}^2+{\sum }_{t=2}^n{\left(Y_t-\mu -\varphi Y_{t-1}\right)}^2$

Step 5: Estimate ${\sigma }_e^2$

The estimate of ${\sigma }_e^2$ is obtained by maximizing the log-likelihood with respect to ${\sigma }_e^2$, treating $\hat{\varphi }$ and $\hat{\mu }$ as already estimated.

Derivation:

From Step 4, the log-likelihood is: $\log L=-\frac{n}{2}\log (2\pi {\sigma }_e^2)+\frac{1}{2}\log (1-{\varphi }^2)-\frac{S(\varphi ,\mu )}{2{\sigma }_e^2}$

Taking the partial derivative with respect to ${\sigma }_e^2$:

$$\begin{array}{rl}\frac{\partial \log L}{\partial {\sigma }_e^2}& =-\frac{n}{2}\cdot \frac{1}{{\sigma }_e^2}+\frac{S(\varphi ,\mu )}{2({\sigma }_e^2{)}^2}\end{array}$$

Setting equal to zero for maximization:

$$\begin{array}{rl}-\frac{n}{2{\sigma }_e^2}+\frac{S(\varphi ,\mu )}{2({\sigma }_e^2{)}^2}& =0\\ \frac{n}{2{\sigma }_e^2}& =\frac{S(\varphi ,\mu )}{2({\sigma }_e^2{)}^2}\\ n& =\frac{S(\varphi ,\mu )}{{\sigma }_e^2}\\ {\hat{\sigma }}_e^2& =\frac{S(\hat{\varphi },\hat{\mu })}{n}\end{array}$$

After obtaining $\hat{\varphi }$ and $\hat{\mu }$ (by maximizing the concentrated likelihood), the estimator is: ${\hat{\sigma }}_e^2=\frac{S(\hat{\varphi },\hat{\mu })}{n}$

Related

• Unconditional Sum-of-Squares Function
• Least Square Method
• Large Sample Properties of Parameter Estimates