Evaluating Forecasts of Return Distributions
Summary
The document asks how to assess a simulated return distribution and compare it with forecasts from another method. Its example uses historical returns to generate simulated paths over a short forecast horizon, then pairs forecasts with subsequent realized observations. Suggested checks include comparing forecast and realized densities or cumulative distributions, examining forecast volatility against realized volatility, and checking whether the mean forecast error is near zero as an indication of bias.
It also proposes two-sample distribution tests, including Kolmogorov–Smirnov and Anderson–Darling variants, and mentions statistical distances such as Kullback–Leibler divergence. These suggestions provide possible comparison tools, but the discussion does not specify a complete validation design or a preferred scoring rule. In particular, a test p-value is not by itself a comprehensive measure of forecast quality, and a proper comparison depends on how the forecast and realized samples are constructed and aligned.
Key ideas
- Compare forecast and realized distributions using their densities or cumulative distribution functions.
- Assess volatility forecasts against realized volatility as a separate diagnostic.
- A mean forecast error near zero can indicate a lack of average directional bias.
- Two-sample tests and statistical distances offer ways to compare distributions.
- A p-value alone does not provide a complete measure of forecasting performance.
Tags
Full text
# estimating the accuracy of a method for forecasting the distribution # estimating the accuracy of a method for forecasting the distribution Say for a stock I want to do a simulation using 30 days of historical returns, and maybe generate 1000 paths, with 2 days as the forecast horizon. Say I have 100 of these 5 day blocks used for generating the distribution, matched with the actual values of the 2 days I am interested in forecasting. How would I estimate the accuracy of this method of generating a distribution? And what is a method I could use to compare it with a different means of forecasting the distribution? Any suggestions would help. ## Answer by phil (score 1) https://quant.stackexchange.com/a/587 Compare the density and cumulative density function or your forecast volatility and of the realized volatility ## Answer by user474 (score 1) https://quant.stackexchange.com/a/592 Seems like you are interested in the forecast error http://en.wikipedia.org/wiki/Forecasting With regard to distribution, I would also look at the mean forecast error. A good model with have a mean error = 0, since it is not bias upward or downward. -Ralph Winters ## Answer by shabbychef (score 0) https://quant.stackexchange.com/a/639 You might want to perform a Two sample Kolmogorov-Smirnov test on the empirical CDFs of the distributions: your forecast and the realized distribution. Then use the p-value of the test as the metric of interest. Other statistical tests of difference in distribution which can be so abused are the Baumgartner, Weiss, Schindler test, and the variations thereupon. I believe there is also a 2-sample Anderson-Darling type test. There are also the non-symmetric Kullback Leibler divergence, and a half-dozen other definitions of statistical distance. Probably you are most limited by the statistical package or programming language you are working with. If using R, I would guess you can find implementations of all of the above.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.