Evaluating Volatility Forecasts Against Realized Variance
Summary
The document outlines ways to compare volatility forecasts, using a GARCH model and a rolling historical estimate as examples. It proposes regressing realized variance on forecast variance, then checking calibration through regression coefficients, examining residuals for structure, and reviewing fit measures such as R-squared and root mean squared error. It also suggests inspecting whether returns scaled by forecast volatility resemble a normal distribution.
Evaluation should match the forecast horizon to the decision being supported, such as the frequency of portfolio rebalancing. The response advises using enough observations and reserving out-of-sample data for assessing fitted model parameters. It also stresses that there is no universal best volatility model: the relevant target can vary with data frequency and intended use, such as risk estimation, option pricing, or predicting a market range. The suggested diagnostics are starting points; the document does not prescribe a single scoring rule or resolve issues in measuring latent volatility.
Key ideas
- Compare forecasts with realized variance using calibration and error diagnostics.
- Inspect residuals for patterns and use fit measures as complementary evidence.
- Check scaled returns with distributional diagnostics such as a quantile plot.
- Evaluate forecasts at the horizon relevant to the trading or risk decision.
- Use adequate samples and out-of-sample data when assessing fitted parameters.
- Choose evaluation targets based on the intended use of the volatility estimate.
Tags
Full text
# How to compare volatility models? # How to compare volatility models? What are the most popular ways to compare volatility models? Suppose I wanted to compare the forecasting accuracy of a GARCH(1,1) model with the historic 30 day volatility. What method should I use? ## Answer by Shane (score 5) https://quant.stackexchange.com/a/9299 There is no one right answer to this question, but a common starting place is to compare the bias and variance of the forecast vs. the realized variance. Take your forecasted variance $\hat y$ and regress them against the realized variance: $y = \beta_0 + \beta_1 \hat y + \epsilon$ A few things that you want to see: - The forecast should be unbiased, meaning that $\beta_1 \approx 0$. If this is not the case, then you will consistently over or under-realize your prediction. - The error should not have any clear structure. So $\epsilon \mapsto N(0, \sigma)$; plot the residuals and look at them. - You can view the goodness-of-fit of your model by looking at things like the $R^2$ and the RMSE. - Normalizing your returns by your forecast should result in a normal distribution: you can review this by using a qqplot. Make sure that you evaluate your model on the time horizon that is relevant to you. For instance, if you're rebalancing monthly, then it won't be especially important to predict the next day's volatility as much as if would if you were rebalancing daily. As with any forecast, make sure that you have a sufficient sample size for significance, and use out-of-sample data to evaluate any parameters that you might fit if you use a structural model. Some useful references: - Hansen and Lunde (2001) "A comparison of volatility models: Does anything beat a GARCH(1,1)?" - Hansen, Lunde, and Nason (2003) "Choosing the Best Volatility Models: The Model Confidence Set Approach" ## Answer by htrahdis (score -1) https://quant.stackexchange.com/a/9305 A model is one which has the ability to make predictions. So use the models you have to make predictions about volatility. The one which can predict the best has to be the best (don't think you have a choice in that). If the predictability is comparable, then use any one. If you cannot test which one is the predicting the best, then go back and study what is volatility. This is the edit after the first comment. For all volatility modelers, please read this paper first We Don't Quite Know What We are Talking About When We Talk About Volatility. Next it is important to understand that the process of volatility modeling is highly dependent on the context in which the volatility is going to be used in. There is no universal right answer. For example, what do you believe in, price volatility or returns volatility and both have their space in financial research. next what time frame are you looking at, is it high frequency data ? or just intraday ? or end of day data. furthermore what do you want to predict using the model, because volatility itself is an invisible variable and hence you have to define some aspect of the market which you want to predict using your volatility model. So it could be to model the risk you face in the next period, or the prediction of the option price or simply the range of the market in the next period and believe me each of this is different. So using the model you would want to predict the measurable aspect of the market and then decide if the volatility model suits you or not. finally you have to understand if the model explains features of a volatility process the best, like regimes in volatility and spillovers to other instruments and the like. This is not a question answered so easily. Keep working.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.