Out-of-Sample GARCH Forecast Evaluation and Diagnostics

# Assessing the GARCH model out-of-time


# Assessing the GARCH model out-of-time












I have fitted two competing GARCH models, one `GARCH(1,2)` model and another `EGARCH(1,1,1)` both with t-distributed errors, on the log-returns of EquinorASA (Norwegian gas equity). I will not go through the preliminary checks, but the assumptions for `GARCH(1,2)` hold.

I have used Kevin Shepard's well-known arch library. To select the model, I would like to split the time-series into train/test (this is easily done with `TimeSeriesSplit` from `sklearn`).

For simplicity, let's just consider the `GARCH(1,2)` model.

This is excerpt from the code:

```
logr_train, logr_test = logr[train_index], logr[test_index]
N_train = len(train_index)
N_test = len(test_index)

specs = {
            'mean':'zero',
            'vol':'GARCH',
            'p':1,
            'q':2,
            'dist':'t',
            'rescale':True
}

model = arch_model(y=logr_train, **specs)
res = model.fit(first_obs=0, last_obs=len(logr_train))
forecast = res.forecast(horizon=N_test, method='simulation', simulations=1000, reindex=False)
```

This is the summary print:

The dilemma: what do we actually compare between train and test?

What I have considered so far?

- Empirical volatility of test log-returns vs. Expectation of simulated volatility This would be simply:

```
forecast.variance.shape # second dimension is the same as size of test time-series
(1, 499) 
np.sqrt(forecast.variance.values[0][-1]) # this is slicing the last value of the simulations (when it converges)
1.6180795243550963
np.std(logr_test) * 100
1.1331922351104446
error = np.sqrt(forecast.variance.values[0][-1]) - np.std(logr_test) * 100
error
0.4848872892446516
```

- Is there a way to somehow recover the log-returns?

Since `GARCH(1,2)` is defined as: $$ X_{t} = \sigma_{t}Z_{t} \\ Z_{t} \sim N(0,1) \\ \sigma_{t}^{2} = \alpha_{0} + \alpha_{1}X^{2}_{t-1} + \beta_{1}\sigma^{2}_{t-1} + \beta_{2}\sigma^{2}_{t-2} $$

Given the estimated parameters, could there be a way to compute predicted $\hat{X_{t+n}}^{2}$ and then compute the error between that and the log-returns of $X_{t+n}$ from the test sample?

I have already checked the Q&As here for "evaluating garch models" and "GARCH model, expectation of volatility?", to no avail.

## Answer by Count (score 2, accepted)

https://quant.stackexchange.com/a/73602

Assume that you observe $T$ returns, then choose some $T_1<T$ and do the following:

- Estimate your model on $r_1,\dots,r_{T_1}$ and use your estimated parameters to predict $\hat{\sigma}_{T_1+1}^2=E(\epsilon_{T_1+1}^2\vert \mathcal F_{T_1})$.

- Estimate your model on $r_1,\dots,r_{T_1+1}$ and use your estimated parameters to predict $\hat{\sigma}_{T_1+2}^2=E(\epsilon_{T_1+2}^2\vert \mathcal F_{T_1+1})$.

$\quad \vdots$

- Estimate your model on $r_1,\dots,r_{T_1+h-1}$ and use your estimated parameters to predict $\hat{\sigma}_{T_1+h}^2=E(\epsilon_{T_1+h}^2\vert \mathcal F_{T_1+h-1})$.

You end up with a time series of $h$ one-step-ahead forecasts. Now, you want to compare these forecasts to the corresponding "true" values of $\sigma_t^2$. However, $\sigma_t^2$ is not directly observable given the fact that you only observe prices or the returns that you can calculate from them. Thus, in order to backtest your model, you have to think about an estimate for $\sigma_t^2$. As it turns out, there are several ways to do that, but it depends largely on the available data.

For example, the easiest approach - but also a problematic one - is to use the squared errors $\epsilon_t^2$ as an estimate for $\sigma_t^2$, that is you compare $\epsilon_{t+h}^2$ with $\hat{\sigma}_{t+h}^2$. Although this estimator is unbiased, it turns out that it is a very noisy estimator of the "true" volatility. Using this estimator then often leads to the conclusion that GARCH models perform poorly out-of-sample. An intuitive explanation why this is the case is the following. Assume that \begin{align} r_t&=\mu_t+\epsilon_t\\ \epsilon_t&=\sigma_tu_t \quad, u_t \overset{iid}{\sim} \mathcal N (0,1) \end{align} then \begin{align} P\left(\epsilon_t^2 \notin \left[\frac{1}{3}\sigma_t^2,\frac{2}{3}\sigma_t^2\right]\right)&=P\left(\sigma_t^2u_t^2 \notin \left[\frac{1}{3}\sigma_t^2,\frac{2}{3}\sigma_t^2\right]\right)\\ &=1-P\left(\sigma_t^2u_t^2 \in \left[\frac{1}{3}\sigma_t^2,\frac{2}{3}\sigma_t^2\right]\right)\\ &=1-P\left(u_t^2 \in \left[\frac{1}{3},\frac{2}{3}\right]\right)\\ &=0.7411 \end{align} Where the last line follows from the fact that $u_t^2 \sim \chi^2(1)$. That is, with a probability of 74% your estimator of $\sigma_t^2$ will lie outside the interval $\left[\frac{1}{3}\sigma_t^2,\frac{2}{3}\sigma_t^2\right]$.

Long story short, your question really boils down to two questions:

- How do I estimate $\sigma_t^2$ in a reasonable sense? The answer is via realized volatility estimators that rely on intraday data. For further details, the papers given below are insightful.

- Which forecast evaluation method should I use? There is no general answer to this, but it depends on the goal of your analysis. For example, you could use RMSE $$ RMSE=\sqrt{\frac{1}{h}\sum_{i=1}^h(\tilde{\sigma}_{t+i}^2-\hat{\sigma}_{t+i}^2)^2} $$ Where $\tilde{\sigma}_{t+i}^2$ is your chosen estimator for $\sigma_t^2$. However, if underprediction is a concern, an asymmetrical loss function would make more sense.

### Further literature

- Poon, Granger 2003: Forecasting Volatility in Financial Markets: A Review, Journal of Economic Literature

- Andersen, Bollerslev, Lange (2001): Forecasting financial market volatility: Sample frequency vis-a-vis forecast horizon, Journal of Empirical Finance

- Andersen, Bollerslev (1998): Answering the Skeptics Yes, Standard Volatility Models do Provide Accurate Forecasts, International Economic Review

## Answer by Richard Hardy (score 2)

https://quant.stackexchange.com/a/73607

A GARCH model is a model not only for the conditional variance but for the entire conditional distribution of a random variable of interest. To assess the model's statistical adequacy out of sample, you would use time series cross validation via rolling or expanding windows as described in Lars's answer. You would then look at one-step-ahead standardized innovations $\hat z_{t+1}$. That is, you would take the actual realization $x_{t+1}$, subtract the predicted conditional mean $\hat\mu_{t+1}$ and then divide the result by the predicted conditional standard deviation $\hat\sigma_{t+1}$: $\hat z_{t+1} := \frac{ x_{t+1} - \hat\mu_{t+1} }{ \hat\sigma_{t+1} }$. These should be i.i.d. with mean=0 and variance=1 and follow the distribution that you have assumed when specifying the model (e.g. Student's $t$ distribution). I do not think there are any tests that would test all of these properties at once. What you would normally do instead is examine them one by one. (Unfortunately, all of the tests I can remember rely on some of these assumptions being true to test a single one of them. Thus, strictly speaking there is no way to test them properly.)

- For i.i.d.'ness, you would look at the autocorrelation function and the partial autocorrelation function of the standardized residuals and their squares. Formal tests could be Ljung-Box (though see this) and Li-Mak (not ARCH-LM on $\hat z_{t+1}$ and not Ljung-Box on $\hat z_{t+1}^2$).

- For the assumed distribution, you would obtain the probability integral transform of the standardized residuals and compare that to a Uniform[0,1] distribution using e.g. Kolmogorov-Smirnov test.

To assess the model's performance in economic or monetary terms, you would have to come up with a loss function to be applied on prediction errors $e_{t+1}=\hat x_{t+1}-x_{t+1}$ made by the model.* (More generally, loss could be defined on the pair of prediction $\hat x_{t+1}$ and the corresponding actual realization $x_{t+1}$ instead of $e_{t+1}$, but it may often suffice to just look at the difference between the two.) You would have to make sure that the point predictions are made in accordance to the evaluation loss function. E.g. if the evaluation loss function is absolute loss, you would not be predicting the conditional mean but rather the conditional median.

For more details on both the statistically adequacy and the economics/monetary performance, check out Elliott & Timmermann "Economic Forecasting" (2016), especially chapters 2, 13 and 18. (I have also written some related answers on Cross Validated; start here and enter additional search phrases to refine the search.)

*In case you are actually directly interested in the conditional variance rather than the realization of the random variable (i.e. the actual return), you would use $(\sigma_{t+1}^2,\hat\sigma_{t+1}^2)$ instead of $(x_{t+1},\hat x_{t+1})$ and define prediction errors accordingly.