Skip to content
All library documents

Evaluating GARCH VaR Backtests and the Limits of MAPE

Article Quant Q&A · Author: Pelumi
Read the original

Summary

The document describes an attempt to assess a GARCH model fitted to S&P 500 returns. It shows a rolling backtest setup and reports difficulty using mean absolute percentage error (MAPE) to compare predicted and realized values. The reply explains that MAPE becomes infinite when an observed value is zero, which can make it unsuitable for return data that may contain zero observations.

The document also presents a VaR backtest report. Its exceedances are higher than expected, and both the Kupiec unconditional coverage test and Christoffersen conditional coverage test reject their null hypotheses. This indicates that, at the reported confidence level, the model’s VaR exceptions are too frequent and their pattern also fails the test’s independence requirement. These diagnostics concern VaR forecasts, while MAPE on the conditional mean addresses a different target. The short exchange does not explain alternative scoring rules, test assumptions, or how to revise the model.

Key ideas

  • MAPE can be infinite when observed values include zero.
  • VaR backtests assess exception frequency and, in some tests, whether exceptions are independent.
  • The reported tests reject correct VaR coverage and independent failures for this model.
  • Forecast accuracy for the conditional mean and VaR calibration are distinct evaluation questions.

Tags

StatisticsVolatilityBacktestingRisk management
Full text
# evaluating garch models


# evaluating garch models












I used ugarchroll to backtest my garch model on S&P returns

this is my code

```
library(rugarch)
library(quantmod)
getSymbols("SPY")

rets = ROC(SPY$SPY.Close, na.pad = FALSE)

tgarch = ugarchspec(mean.model = list(armaOrder = c(1, 1)), 
                    variance.model = list(model = "sGARCH"),
                    distribution.model = "std")

garchroll <- ugarchroll(tgarch, data=rets, n.start=500, 
                        refit.window="window", refit.every=200)
```

however I am having trouble evaluating my model backtest . I tried to evaluate my model using MAPE - this was the code I used to get the MAPE OF my backtest

```
library(forecast)
preds<-as.data.frame(garchroll)

accuracy(preds$Mu, preds$Realized)
```

however When I tried to get my MAPE got

```
Inf
```

I also tried to use the report function to evaluate my model

```
report(garchroll)
```

however I do not know how to interpret the results of my model

```
VaR Backtest Report
===========================================
Model:              sGARCH-std
Backtest Length:    2719
Data:               

==========================================
alpha:              1%
Expected Exceed:    27.2
Actual VaR Exceed:  50
Actual %:           1.8%

Unconditional Coverage (Kupiec)
Null-Hypothesis:    Correct Exceedances
LR.uc Statistic:    15.491
LR.uc Critical:     3.841
LR.uc p-value:      0
Reject Null:        YES

Conditional Coverage (Christoffersen)
Null-Hypothesis:    Correct Exceedances and
                    Independence of Failures
LR.cc Statistic:    16.486
LR.cc Critical:     5.991
LR.cc p-value:      0
Reject Null:        YES
```

please help me interpret the results of my garch model your help will be greatly appreciated

## Answer by Psi (score 1)

https://quant.stackexchange.com/a/49308

If your label data contains any zeroes, the MAPE of any prediction when the label is 0 is infinite...

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.