Photo by Annie Spratt on Unsplash
An autoregressive (AR) model predicts the future value based on previous values. Before jumping into the math behind AR models, we need to discuss a concept called stationarity.
Stationarity
A time series is stationary if its properties are independent of the times at which the time series is observed. For e.g. the properties of the stationary time series ($r_{t_1}, …, r_{t_k}$) is identical to that of ($r_{t_1+1}, …, r_{t_k+1}$) for all $t$.
More specifically, a time series ($r_{1}, r_{2}, …, r_{t}$) is stationary if
\[E[r_t] = \mu\]for all $t$,
\[\text{Cov}(r_{t}, r_{r-l})=\gamma_{l}\]which only depends on $l$.
Autoregressive Models
Now lets get back to AR models. An AR($p$) is written as follows
\[y_{t} = \phi_{0} + \phi_{1}y_{t-1} + \phi_{2}y_{t-2} + ... + \phi_{p}y_{t-p} + \epsilon_{t}\]where $\epsilon_{t}$ is white noise with E[$\epsilon_{t}$] = 0 and Var[$\epsilon_{t}$] = $\sigma^2$
AR(1) Model
Now lets look at AR(1) model in detail
\[\begin{align} \label{eqn:ar1} y_{t} = \phi_{0} + \phi_{1}y_{t-1} + \epsilon_{t} \end{align}\]Taking the expectation of this equation
\[\begin{align} E[y_{t}] &= E[\phi_{0} + \phi_{1}y_{t-1} + \epsilon_{t}] \nonumber \\ \mu &= E[\phi_{0}] + E[\phi_{1}y_{t-1}] + E[\epsilon_{t}] \nonumber \\ &= E[\phi_{0}] + \phi_{1}E[y_{t-1}] + E[\epsilon_{t}] \nonumber \\ &= \phi_{0} + \phi_{1}\mu \end{align}\]Solving for $\mu$ we get
\[\begin{align} \label{eqn:mu_ar1} \mu = \frac{\phi_{0}}{1-\phi_{1}} \end{align}\]Now, lets take the variance of Eq. \ref{eqn:ar1}
\[\begin{align} \text{Var}[y_{t}] &= \text{Var}[\phi_{0} + \phi_{1}y_{t-1} + \epsilon_{t}] \nonumber \\ &= \text{Var}[\phi_{0}] + \text{Var}[\phi_{1}y_{t-1}] + \text{Var}[\epsilon_{t}] \nonumber \\ &= \phi_{1}^2\text{Var}[y_{t-1}] + \sigma^{2} \end{align}\]using the property of stationarity ($\text{Var}[y_{t}] =\text{Var}[y_{t-1}]$)
\[\text{Var}[y_{t}] = \frac{\sigma^{2}}{1 - phi_{1}^2}\]notice that $\phi_{1}^2$ < 1, because varaince needs to finite and is by definition nonnegative.
We can rearrange Eq. \ref{eqn:mu_ar1} to get
\[\phi_{0} = (1 - \phi_{1})\mu\]and rewrite Eq. \ref{eqn:ar1} as
\[\begin{align} y_{t} &= \phi_{0} + \phi_{1}y_{t-1} + \epsilon_{t} \nonumber \\ &= (1 - \phi_{1})\mu + \phi_{1}y_{t-1} + \epsilon_{t} \end{align}\]We can rewrite the equation above as
\[\begin{align} y_{t} - \mu &= -\phi_{1}\mu + \phi_{1}y_{t-1} + \epsilon_{t} \nonumber \\ &= \phi_{1}(y_{t-1} - \mu) + \epsilon_{t} \end{align}\]Autocorrelation Fucntion of AR(1) Model
The correlation between two random variable $X$ and $Y$ is defined as
\[\begin{align} \rho_{x,y} &= \frac{\text{Cov}[X,Y]}{\sqrt{\text{Var}[X] \text{Var}[Y] }} \nonumber \\ &= \frac{ E[(X-\mu_{x}) (Y-\mu_{y}) )] }{\sqrt{ E[(X-\mu_{x})^2] E[(Y-\mu_{y})^2] }} \end{align}\]Let
\[\begin{align} \rho_{l} &= \frac{\text{Cov}[y_{t},y_{t-l}]}{\sqrt{\text{Var}[y_{l}] \text{Var}[y_{t-l}] } } \nonumber \\ &= \frac{\gamma_{l}}{\gamma_{0}} \end{align}\]where $\text{Var}[y_{l}]=\text{Var}[y_{t-l}]$ because of stationarity and $l$ is the lag.
\[\begin{align} \gamma_{l} &= E[ (y_{t}-\mu)(y_{t-l} - \mu) ] \nonumber \\ &= E[\phi_{1}(y_{t-1}-\mu)(y_{t-l} - \mu) + \epsilon_{t}(y_{t-l} - \mu)] \nonumber \\ &= \phi_{1}E[(y_{t-1}-\mu)(y_{t-l} - \mu)] + E[\epsilon_{t}(y_{t-l} - \mu)] \nonumber \\ &= \phi_{1}\gamma_{l-1} + E[\epsilon_{t}(y_{t-l} - \mu)] \end{align}\]if $l$=0
\[\begin{align} E[\epsilon_{t}(y_{t} - \mu)] &= E[\epsilon_{t} ( (\phi_{1}(y_{t-1} - \mu) + \epsilon_{t}) ) ] \nonumber \\ &= E[\epsilon_{t}(\phi_{1}(y_{t-1} - \mu)] + E[\epsilon_{t}^2] \nonumber \\ &= \sigma^2 \end{align}\]if $l$>0
\[\begin{align} E[\epsilon_{t}(y_{t-l} - \mu)] &= E[\epsilon_{t} ( (\phi_{1}(y_{t-l-1} - \mu) + \epsilon_{t-l}) ) ] \nonumber \\ &= 0 \end{align}\]Therefore,
\[\begin{align} \gamma_{l} &= \begin{cases} \phi_{1} \gamma_{1} + \sigma^2,~~~~ l=0 \\ \phi_{1} \gamma_{l-1},~~~~ l>0 \end{cases} \end{align}\]and
\[\begin{align} \rho_{l} &= \begin{cases} 1,~~~~ l=0 \\ \phi_{1}\rho_{l-1},~~~~ l>0 \end{cases} \end{align}\]For $\rho > 0$, the ACF of a AR(1) series decays exponentially with rate $\phi_1$. While for a negative $\rho$, the plot consists of 2 alternating exponential decays with rate $\phi_{1}^2$.
References:
- Tsay, Ruey S. Analysis of financial time series, 3rd edition. John wiley & sons, 2010.
Comments