2 Fundamental Concepts

This note covers stochastic processes, and their core concepts:

• Mean
• Covariance functions
• Stationary
• Autocorrelation

---

About time series and stochastic processes

A stochastic process is a sequence of random variables $\{Y_t:t\in \mathbb{Z}\}$, used to model time series data.

Its full probabilistic structure is defined by the joint distributions of all finite subsets of $Y_t$.

In practice, we focus on means, variances, and covariances (first and second moments) rather than the full distributions.

Means, variances, and covariances

For a stochastic process $\{Y_t\}$, its key functions are:

• Mean function ${\mu }_t=E(Y_t)$ The expected value at time $t$.
• Autocovariance function ${\gamma }_{t,s}=\mathrm{Cov}(Y_t,Y_s)=E[(Y_t-{\mu }_t)(Y_s-{\mu }_s)]$ Measuring dependence between $Y_t$ and $Y_s$.
• Autocorrelation function ${\rho }_{t,s}=\mathrm{Corr}(Y_t,Y_s)=\frac{{\gamma }_{t,s}}{\sqrt{{\gamma }_{t,t}{\gamma }_{s,s}}}$ Unitless measure of linear dependence.

Properties

• ${\gamma }_{t,t}=\mathrm{Var}(Y_t)$
• $|{\rho }_{t,s}|\le 1$
• ${\rho }_{t,t}=1$
• ${\gamma }_{t,s}={\gamma }_{s,t},{\rho }_{t,s}={\rho }_{s,t}$ (symmetry)

A key result for covariance of linear combinations is:

$$\mathrm{Cov}\left(\sum _{i=1}^m{c}_i{Y}_{t_i},\sum _{j=1}^n{d}_j{Y}_{s_j}\right)=\sum _{i=1}^m\sum _{j=1}^n{c}_i{d}_j\mathrm{Cov}(Y_{t_i},Y_{s_j})$$

Some stochastic processes

Suppose that

• $\{e_t\}$ are independent, identically distributed (i.i.d.)
• $E(e_t)=0$
• $\mathrm{Var}(e_t)={\sigma }_e^2$

Random walk

$$\begin{array}{rl}{Y}_t& =e_1+e_2+\cdots +e_t\\ & =\sum _{i=1}^t{e}_i\end{array}$$

Property	Expression
Mean	$E(Y_t)=0$
Variance	$\mathrm{Var}(Y_t)=t{\sigma }_e^2$
Autocovariance	${\gamma }_{t,s}=t{\sigma }_e^{2}1\le t\le s$
Autocorrelation	${\rho }_{t,s}=\sqrt{\frac{t}{s}}1\le t\le s$

Notice the variance of the process increases with time.

Random walk also has the inductive form:

$$\begin{array}{r}{Y}_t=Y_{t-1}+e_t\end{array}$$

With “initial state” $Y_1=e_1$.

Intuition

If the $e_t$ is interpreted as the size of “step” taken at time $t$, then $Y_t$ can be interpreted as the position of the “random walker” at time $t$.

Also notice the following autororrelation values


Values of $Y$ at neighboring time points are more and more strongly and positively correlated as time goes by.

On the other hand, the values of $Y$ at distant time points are less and less correlated.

Code example:

import random
 
random_walk: list[float] = [0.0]
for _ in range(100):
    random_walk.append(random_walk[-1] + random.uniform(-1,1))
 
import matplotlib.pyplot as plt
plt.plot(random_walk)
plt.show()

Moving average

$$Y_t=\frac{e_t+e_{t-1}}{2}$$

Property	Expression
Mean	$E(Y_t)=0$
Variance	$\mathrm{Var}(Y_t)=0.5{\sigma }_e^2$
Autocovariance	${\gamma }_{t,s}=\{\begin{array}{ll}0.5{\sigma }_e^2& ,|t-s|=0\\ 0.25{\sigma }_e^2& ,|t-s|=1\\ 0& ,|t-s|>1\end{array}$
Autocorrelation	${\rho }_{t,s}=\{\begin{array}{ll}1& ,|t-s|=0\\ 0.5& ,|t-s|=1\\ 0& ,|t-s|>1\end{array}$

Notice that values of $Y$ at $k$ units of time apart have the same correlation no matter where they occur (for any $t,s$).

This leads us to an important concept in time series which is stationarity, which will be covered in Stationarity.