4 Models for Stationary Time Series

This note covers autoregressive (AR) and moving average (MA) models, focusing on their statistical properties, stationarity, and invertibility.

It also covers general linear processes, moving average (MA) processes, autoregressive (AR) processes, and mixed ARMA models.

---

General stochastic linear processes

Let

• $t\in N$ : Time index
• $Y_t$ : Process value at time $t$
• $e_t$ : White noise error term at time t t t, i.i.d. with mean 0 and constant variance.

A general linear process is:

$Y_t=e_t+{\psi }_1{e}_{t-1}+{\psi }_2{e}_{t-2}+\cdots$

Assuming ${\psi }_0=1$, the mean is $E(Y_t)=0$, and autocovariance is: ${\gamma }_k={\sigma }_e^2{\sum }_{i=0}^{\infty }{\psi }_i{\psi }_{i+k},k\ge 0$

It is convergent if ${\sum }_{i=1}^{\infty }{\psi }_i^2<\infty$.

• Example:

  Suppose ${\psi }_j={\varphi }^j$ and $|\varphi |<1$.

  Then, $\mathrm{Var}(Y_t)=\frac{{\sigma }_e^2}{1-{\varphi }^2},\mathrm{Corr}(Y_t,Y_{t-k})={\varphi }^k$

  This process is stationary, with autocovariance depending only on lag.

Moving average (MA) processes

An MA($q$) process has finite nonzero $\psi$-weights:

$Y_t=e_t-{\sum }_{i=1}^q{\theta }_i{e}_{t-i}$ where:

• $q\in \mathbb{N}$ : Order of the model. Number of error terms
• $i\in \mathbb{N}$ : Time lag
• ${\theta }_i$ : Model parameter for lag $i$
• $e_{t-i}$ : Lagged error; error at time $t-i$

These models are called short memory models, since the errors doesn’t last long into the future. To illustrate:


This goes back to the idea of stationarity, where the dependence of previous observations “declines” over time, or in the case of MA models, actually disappear completely as you go into the future.

MA(1) Process

$Y_t=e_t-\theta e_{t-1}$

This is a model that depends only on one lag of error in the past.

Property	Expression
Mean	$E(Y_t)=0$
Variance	${\gamma }_0={\sigma }_e^2(1+{\theta }^2)$
Covariance	${\gamma }_1=-\theta {\sigma }_e^2$
Autocorrelation	${\rho }_1=\frac{-\theta }{1+{\theta }^2}$, ${\rho }_k=0,k\ge 2$

${\rho }_1$ ranges from $-0.5$ to $0.5$, at $\theta =-1$ to $1$ respectively.

Simulations show positive $\theta$ yields jagged series, negative $\theta$ smoother series.

MA(2) Process

$Y_t=e_t-{\theta }_1{e}_{t-1}-{\theta }_2{e}_{t-2}$

Property	Expression
Variance	${\gamma }_0=(1+{\theta }_1^2+{\theta }_2^2){\sigma }_e^2$
Covariance	\begin{align} \gamma_1 &= (-\theta_1 + \theta_1 \theta_2) \sigma_e^2 \\ \gamma_2 &= -\theta_2 \sigma_e^2 \end{align}
Autocorrelation	\begin{align} \rho_1 &= \frac{-\theta_1 + \theta_1 \theta_2}{1 + \theta_1^2 + \theta_2^2} \\ \rho_2 &= \frac{-\theta_2}{1 + \theta_1^2 + \theta_2^2} \\ \rho_k &= 0, k \geq 3 \end{align}

General MA($q$)

Property	Expression
Variance	${\gamma }_0=\left(1+{\sum }_{i=1}^q{\theta }_i^2\right){\sigma }_e^2$
Covariance
Autocorrelation	\begin{align} \rho_1 &= \frac{-\theta_1 + \theta_1 \theta_2}{1 + \theta_1^2 + \theta_2^2} \\ \rho_2 &= \frac{-\theta_2}{1 + \theta_1^2 + \theta_2^2} \\ \rho_k &= 0, k \geq 3 \end{align}


See also

• https://www.youtube.com/watch?v=zNLG8tsA_Go

Autoregressive (AR) processes

An AR($p$) process satisfies:

$Y_t=e_t+{\sum }_{i=1}^p{\varphi }_i{Y}_{t-i}$ where:

• $p\in \mathbb{N}$ : Order of the model. Number of recursions.
• ${\varphi }_i$ : Model parameter for lag $i$
• $e_t$: Error at time $t$

In contrast to the moving average model, in AR models each observation depends on all previous observation recursively.

AR(1) Process

For $Y_t=\varphi Y_{t-1}+e_t$:

Property	Expression
Variance	${\gamma }_0=\frac{{\sigma }_e^2}{1-{\varphi }^2}$
Autocovariance	${\gamma }_k={\varphi }^k\frac{{\sigma }_e^2}{1-{\varphi }^2}$
Autocorrelation	${\rho }_k={\varphi }^k$

• $|\varphi |<1$ → $\{Y_t\}$ is stationarity:

AR(2) Process

$Y_t={\varphi }_1{Y}_{t-1}+{\varphi }_2{Y}_{t-2}+e_t$

• Stationarity requires roots of $1-{\varphi }_{1}x-{\varphi }_2{x}^2=0$ to exceed 1 in modulus, satisfied by:

${\varphi }_1+{\varphi }_2<1,{\varphi }_2-{\varphi }_1<1,|{\varphi }_2|<1$

• Autocorrelation follows Yule-Walker equation:

${\rho }_k={\varphi }_1{\rho }_{k-1}+{\varphi }_2{\rho }_{k-2},k\ge 1$

• Initial values:

$$\begin{array}{rl}{\rho }_1& =\frac{{\varphi }_1}{1-{\varphi }_2}\\ {\rho }_2& =\frac{{\varphi }_2(1-{\varphi }_2)+{\varphi }_1^2}{1-{\varphi }_2}\end{array}$$

• Variance:

${\gamma }_0=\left(\frac{1-{\varphi }_2}{1+{\varphi }_2}\right)\frac{{\sigma }_e^2}{(1-{\varphi }_2{)}^2-{\varphi }_1^2}$

General AR($p$)

Autocorrelation:

${\rho }_k={\varphi }_1{\rho }_{k-1}+\cdots +{\varphi }_p{\rho }_{k-p},k\ge 1$

Variance: ${\gamma }_0=\frac{{\sigma }_e^2}{1-{\varphi }_1{\rho }_1-\cdots -{\varphi }_p{\rho }_p}$.

Autoregressive moving average (ARMA) process

An ARMA($p$,$q$) model is:

$Y_t=e_t+{\sum }_{i=1}^p{\varphi }_i{Y}_{t-i}-{\sum }_{i=1}^q{\theta }_i{e}_{t-i}$

ARMA(1,1) Model

For $Y_t=\varphi Y_{t-1}+e_t-\theta e_{t-1}$, $|\varphi |<1$ ensures stationarity:

Property	Expression
Variance	${\gamma }_0=\frac{1-2\varphi \theta +{\theta }^2}{1-{\varphi }^2}{\sigma }_e^2$
Autocovariance	\begin{align} \gamma_1 &= \phi \gamma_0 - \theta \sigma_e^2 \\ \gamma_k &= \phi \gamma_{k-1}, \quad k \geq 2 \end{align}
Autocorrelation	${\rho }_k=\frac{(1-\theta \varphi )(\varphi -\theta )}{1-2\theta \varphi +{\theta }^2}{\varphi }^{k-1},k\ge 1$

General ARMA($p$,$q$)

Stationarity requires AR roots to exceed 1 in modulus. Autocorrelation satisfies ${\rho }_k={\varphi }_1{\rho }_{k-1}+\cdots +{\varphi }_p{\rho }_{k-p}$ for $k>q$.

Invertibility

An MA($q$) process is invertible if it can be written as an infinite AR process, requiring roots of $1-{\theta }_{1}x-\cdots -{\theta }_q{x}^q=0$ to exceed 1 in modulus.

An MA(1) is invertible, if $|\theta |<1$ for all of its parameters.

For further reading, see Invertability in Time Series Models.